A Mixed-Reality System for Breast Surgical Planning

One quarter of women who undergo breast lumpectomy to treat early-stage breast cancer in the United States undergo a repeat surgery due to concerns that residual tumor was left behind. This has led to a significant increase in women choosing mastectomy operations in the United States. We have developed a mixed-reality system that projects a 3D “hologram” of images from a breast MRI onto a patient using the Microsoft HoloLens. The goal of this system is to reduce the number of repeated surgeries by improving surgeons’ ability to determine tumor extent. We are conducting a pilot study in patients with palpable tumors that tests a surgeon’s ability to accurately identify the tumor location via mixed-reality visualization during surgical planning. Although early results are promising, it is critical but not straightforward to align holograms to the breast and to account for tissue deformations. More work is needed to improve the registration and holographic display at arm’s-length working distance. Nonetheless, first results from breast cancer surgeries have shown that mixed-reality guidance can indeed provide information about tumor location, and that this exciting new use for AR has the potential to improve the lives of many patients.

[1]  Francesco Sardanelli,et al.  Contrast enhanced breast MRI: spatial displacement from prone to supine patient's position. Preliminary results. , 2012, European journal of radiology.

[2]  Brittany Lapin,et al.  Repeat surgery after breast conservation for the treatment of stage 0 to II breast carcinoma: a report from the National Cancer Data Base, 2004-2010. , 2014, JAMA surgery.

[3]  T M Peters,et al.  Virtual and Augmented Medical Imaging Environments: Enabling Technology for Minimally Invasive Cardiac Interventional Guidance , 2010, IEEE Reviews in Biomedical Engineering.

[4]  Frank Sauer,et al.  An augmented reality system for MR image-guided needle biopsy: initial results in a swine model. , 2006, Radiology.

[5]  N. Navab,et al.  Advanced Medical Displays: A Literature Review of Augmented Reality , 2008, Journal of Display Technology.

[6]  J O Barentsz,et al.  Breast tumors: comparative accuracy of MR imaging relative to mammography and US for demonstrating extent. , 1995, Radiology.

[7]  Isabelle Bedrosian,et al.  Disparities in the Use of Breast-Conserving Therapy Among Patients With Early-Stage Breast Cancer. , 2015, JAMA surgery.

[8]  M. A. Lago,et al.  Breast prone-to-supine deformation and registration using a Time-of-Flight camera , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[9]  Yoshikazu Nakajima,et al.  Image guidance of breast cancer surgery using 3-D ultrasound images and augmented reality visualization , 1998, IEEE Transactions on Medical Imaging.

[10]  Thomas E. Yankeelov,et al.  Realization of a biomechanical model-assisted image guidance system for breast cancer surgery using supine MRI , 2015, International Journal of Computer Assisted Radiology and Surgery.

[11]  Brian A Hargreaves,et al.  MR imaging of magnetic ink patterns via off‐resonance sensitivity , 2018, Magnetic resonance in medicine.

[12]  Brian P. Baillargeon,et al.  Modeling Pathologies of Diastolic and Systolic Heart Failure , 2015, Annals of Biomedical Engineering.

[13]  C. Kuhl.  The current status of breast MR imaging. Part I. Choice of technique, image interpretation, diagnostic accuracy, and transfer to clinical practice. , 2007, Radiology.

[14]  Nassir Navab,et al.  Single-Point Active Alignment Method (SPAAM) for Optical See-Through HMD Calibration for Augmented Reality , 2002, Presence: Teleoperators & Virtual Environments.

[15]  E. McVeigh,et al.  Interventional cardiovascular procedures guided by real‐time MR imaging: An interactive interface using multiple slices, adaptive projection modes and live 3D renderings , 2007, Journal of magnetic resonance imaging : JMRI.

[16]  P. Prorok,et al.  Breast-Cancer Tumor Size, Overdiagnosis, and Mammography Screening Effectiveness. , 2016, The New England journal of medicine.

[17]  Ramesh Raskar,et al.  Augmented Reality Visualization for Laparoscopic Surgery , 1998, MICCAI.

[18]  T. J. Eberlein.  Breast-Conserving Surgery Using Projection and Reproduction Techniques of Surgical-Position Breast MRI in Patients with Ductal Carcinoma In Situ of the Breast , 2009 .

[19]  Mehrdad Heydarzadeh,et al.  Augmented reality for breast tumors visualization , 2016, 2016 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[20]  C. Sietses,et al.  A nationwide pathology study on surgical margins and excision volumes after breast-conserving surgery: There is still much to be gained. , 2016, Breast.

[21]  Nassir Navab,et al.  Camera Augmented Mobile C-Arm (CAMC): Calibration, Accuracy Study, and Clinical Applications , 2010, IEEE Transactions on Medical Imaging.

[22]  D M Ikeda,et al.  Three-dimensional shaded-surface rendering of MR images of the breast: technique, applications, and impact on surgical management of breast disease. , 1998, Radiographics : a review publication of the Radiological Society of North America, Inc.

[23]  M Wendt,et al.  [A head-mounted display system for augmented reality: initial evaluation for interventional MRI]. , 2003, RoFo : Fortschritte auf dem Gebiete der Rontgenstrahlen und der Nuklearmedizin.

[24]  Guido Gerig,et al.  User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability , 2006, NeuroImage.

[25]  Jong-Hyeon Jeong,et al.  Twenty-year follow-up of a randomized trial comparing total mastectomy, lumpectomy, and lumpectomy plus irradiation for the treatment of invasive breast cancer. , 2002, The New England journal of medicine.

[26]  T. Peters,et al.  Intraoperative ultrasound for guidance and tissue shift correction in image-guided neurosurgery. , 2000, Medical physics.

[27]  Gudrun Klinker,et al.  Interaction-free calibration for optical see-through head-mounted displays based on 3D Eye localization , 2014, 2014 IEEE Symposium on 3D User Interfaces (3DUI).

[28]  Francisco José Madrid-Cuevas,et al.  Automatic generation and detection of highly reliable fiducial markers under occlusion , 2014, Pattern Recognit..

[29]  S. Kabus,et al.  Symmetric Biomechanically Guided Prone-to-Supine Breast Image Registration , 2015, Annals of Biomedical Engineering.

[30]  David J. Hawkes,et al.  MR Navigated Breast Surgery: Method and Initial Clinical Experience , 2008, MICCAI.

[31]  A. Jemal,et al.  Cancer statistics, 2016 , 2016, CA: a cancer journal for clinicians.

[32]  Nassir Navab,et al.  On mixed reality environments for minimally invasive therapy guidance: Systems architecture, successes and challenges in their implementation from laboratory to clinic , 2013, Comput. Medical Imaging Graph..

[33]  L. Philpotts,et al.  Breast ultrasonography: state of the art. , 2013, Radiology.

[34]  Shinji Naganawa,et al.  Prediction of prone-to-supine tumor displacement in the breast using patient position change: investigation with prone MRI and supine CT , 2014, Breast Cancer.

[35]  Mehran Ebrahimi,et al.  Using surface markers for MRI guided breast conserving surgery: a feasibility survey , 2014, Physics in medicine and biology.

[36]  Keith D. Paulsen,et al.  Supine Breast MRI and 3D Optical Scanning: A Novel Approach to Improve Tumor Localization for Breast Conserving Surgery , 2014, Annals of Surgical Oncology.

[37]  Philippe Young,et al.  The mechanics of decompressive craniectomy: Personalized simulations , 2017 .

[38]  K. Palmer,et al.  Role of image-guidance systems during NOTES. , 2009, Journal of endourology.

[39]  Rebecca L. Siegel Mph,et al.  Cancer statistics, 2016 , 2016 .