3D Registration of pre-surgical prostate MRI and histopathology images via super-resolution volume reconstruction

The use of MRI for prostate cancer diagnosis and treatment is increasing rapidly. However, identifying the presence and extent of cancer on MRI remains challenging, leading to high variability in detection even among expert radiologists. Improvement in cancer detection on MRI is essential to reducing this variability and maximizing the clinical utility of MRI. To date, such improvement has been limited by the lack of accurately labeled MRI datasets. Data from patients who underwent radical prostatectomy enables the spatial alignment of digitized histopathology images of the resected prostate with corresponding pre-surgical MRI. This alignment facilitates the delineation of detailed cancer labels on MRI via the projection of cancer from histopathology images onto MRI. We introduce a framework that performs 3D registration of whole-mount histopathology images to pre-surgical MRI in three steps. First, we developed a novel multi-image super-resolution generative adversarial network (miSRGAN), which learns information useful for 3D registration by producing a reconstructed 3D MRI. Second, we trained the network to learn information between histopathology slices to facilitate the application of 3D registration methods. Third, we registered the reconstructed 3D histopathology volumes to the reconstructed 3D MRI, mapping the extent of cancer from histopathology images onto MRI without the need for slice-to-slice correspondence. When compared to interpolation methods, our super-resolution reconstruction resulted in the highest PSNR relative to clinical 3D MRI (32.15 dB vs 30.16 dB for BSpline interpolation). Moreover, the registration of 3D volumes reconstructed via super-resolution for both MRI and histopathology images showed the best alignment of cancer regions when compared to (1) the state-of-the-art RAPSODI approach, (2) volumes that were not reconstructed, or (3) volumes that were reconstructed using nearest neighbor, linear, or BSpline interpolations. The improved 3D alignment of histopathology images and MRI facilitates the projection of accurate cancer labels on MRI, allowing for the development of improved MRI interpretation schemes and machine learning models to automatically detect cancer on MRI.

[1]  Anant Madabhushi,et al.  Co-Registration of ex vivo Surgical Histopathology and in vivo T2 weighted MRI of the Prostate via multi-scale spectral embedding representation , 2017, Scientific Reports.

[2]  Konstantinos Kamnitsas,et al.  Multi-input Cardiac Image Super-Resolution Using Convolutional Neural Networks , 2016, MICCAI.

[3]  Verónica Vilaplana,et al.  Brain MRI super-resolution using 3D generative adversarial networks , 2018, ArXiv.

[4]  Aaron Fenster,et al.  Prostate: registration of digital histopathologic images to in vivo MR images acquired by using endorectal receive coil. , 2012, Radiology.

[5]  Richard E. Fan,et al.  Prostate Magnetic Resonance Imaging Interpretation Varies Substantially Across Radiologists. , 2017, European urology focus.

[6]  Jeff Wood,et al.  Super‐resolution musculoskeletal MRI using deep learning , 2018, Magnetic resonance in medicine.

[7]  Christian Ledig,et al.  Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Baris Turkbey,et al.  Multiparametric 3T prostate magnetic resonance imaging to detect cancer: histopathological correlation using prostatectomy specimens processed in customized magnetic resonance imaging based molds. , 2011, The Journal of urology.

[9]  P. Humphrey,et al.  Histopathology of Prostate Cancer. , 2017, Cold Spring Harbor perspectives in medicine.

[10]  Dinggang Shen,et al.  Convolutional Neural Network for Reconstruction of 7T-like Images from 3T MRI Using Appearance and Anatomical Features , 2016, LABELS/DLMIA@MICCAI.

[11]  Cheng Soon Ong,et al.  Development of a registration framework to validate MRI with histology for prostate focal therapy. , 2015, Medical physics.

[12]  Luis Ibáñez,et al.  The Design of SimpleITK , 2013, Front. Neuroinform..

[13]  Wei Shao,et al.  ProsRegNet: A deep learning framework for registration of MRI and histopathology images of the prostate , 2020, Medical Image Anal..

[14]  Kathryn J Fowler,et al.  Accuracy and Variability of Prostate Multiparametric Magnetic Resonance Imaging Interpretation Using the Prostate Imaging Reporting and Data System: A Blinded Comparison of Radiologists. , 2020, European urology focus.

[15]  M. Haider,et al.  The expanding role of MRI in prostate cancer. , 2013, AJR. American journal of roentgenology.

[16]  Arvid Lundervold,et al.  Intensity-based volumetric registration of magnetic resonance images and whole-mount sections of the prostate , 2017, Comput. Medical Imaging Graph..

[17]  Debiao Li,et al.  Efficient and Accurate MRI Super-Resolution using a Generative Adversarial Network and 3D Multi-Level Densely Connected Network , 2018, MICCAI.

[18]  Aaron Carass,et al.  Self super-resolution for magnetic resonance images using deep networks , 2018, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[19]  R. Lenkinski,et al.  Elastic registration of multimodal prostate MRI and histology via multiattribute combined mutual information. , 2011, Medical physics.

[20]  Hyunjin Park,et al.  Registration methodology for histological sections and in vivo imaging of human prostate. , 2008, Academic radiology.

[21]  Anant Madabhushi,et al.  HistoStitcher©: An interactive program for accurate and rapid reconstruction of digitized whole histological sections from tissue fragments , 2011, Comput. Medical Imaging Graph..

[22]  Nicolas Passat,et al.  Multiscale brain MRI super-resolution using deep 3D convolutional networks , 2019, Comput. Medical Imaging Graph..

[23]  Wei Shao,et al.  Registration of presurgical MRI and histopathology images from radical prostatectomy via RAPSODI , 2020, Medical physics.

[24]  Mirabela Rusu,et al.  An Application of Generative Adversarial Networks for Super Resolution Medical Imaging , 2018, 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA).

[25]  Jian Zhang,et al.  Fusing multi-scale information in convolution network for MR image super-resolution reconstruction , 2018, BioMedical Engineering OnLine.

[26]  Anant Madabhushi,et al.  Spatially weighted mutual information (SWMI) for registration of digitally reconstructed ex vivo whole mount histology and in vivo prostate MRI , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[27]  Mirabela Rusu,et al.  Anisotropic Super Resolution In Prostate Mri Using Super Resolution Generative Adversarial Networks , 2019, 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019).

[28]  G. Metzger,et al.  Registration of in vivo prostate MRI and pseudo‐whole mount histology using Local Affine Transformations guided by Internal Structures (LATIS) , 2015, Journal of magnetic resonance imaging : JMRI.

[29]  Stephen M. Moore,et al.  The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository , 2013, Journal of Digital Imaging.

[30]  Yu Yang,et al.  Simultaneous single- and multi-contrast super-resolution for brain MRI images based on a convolutional neural network , 2018, Comput. Biol. Medicine.

[31]  P Puech,et al.  Detecting prostate cancer with MRI - why and how. , 2012, Diagnostic and interventional imaging.

[32]  Li Fei-Fei,et al.  Perceptual Losses for Real-Time Style Transfer and Super-Resolution , 2016, ECCV.

[33]  Chi-Hieu Pham,et al.  Brain MRI super-resolution using deep 3D convolutional networks , 2017, 2017 IEEE 14th International Symposium on Biomedical Imaging (ISBI 2017).

[34]  Guido Gerig,et al.  User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability , 2006, NeuroImage.

[35]  Ziv Yaniv,et al.  SimpleITK Image-Analysis Notebooks: a Collaborative Environment for Education and Reproducible Research , 2017, Journal of Digital Imaging.

[36]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[37]  Holden H. Wu,et al.  A system using patient‐specific 3D‐printed molds to spatially align in vivo MRI with ex vivo MRI and whole‐mount histopathology for prostate cancer research , 2018, Journal of magnetic resonance imaging : JMRI.

[38]  L. Klotz,et al.  Prostate cancer overdiagnosis and overtreatment , 2013, Current opinion in endocrinology, diabetes, and obesity.

[39]  Hans Garmo,et al.  Radical prostatectomy or watchful waiting in early prostate cancer. , 2014, The New England journal of medicine.

[40]  Hongming Shan,et al.  MRI Super-Resolution With Ensemble Learning and Complementary Priors , 2020, IEEE Transactions on Computational Imaging.

[41]  A. Rosenkrantz,et al.  Multiparametric MRI for prostate cancer diagnosis: current status and future directions , 2019, Nature Reviews Urology.

[42]  Gözde B. Ünal,et al.  Diffusion MRI Spatial Super-Resolution Using Generative Adversarial Networks , 2018, PRIME@MICCAI.

[43]  Shihui Ying,et al.  Super-resolution reconstruction of MR image with a novel residual learning network algorithm , 2018, Physics in medicine and biology.

[44]  Indrani Bhattacharya,et al.  CorrSigNet: Learning CORRelated Prostate Cancer SIGnatures from Radiology and Pathology Images for Improved Computer Aided Diagnosis , 2020, MICCAI.