The Impact of Machine Learning on 2D/3D Registration for Image-Guided Interventions: A Systematic Review and Perspective

Image-based navigation is widely considered the next frontier of minimally invasive surgery. It is believed that image-based navigation will increase the access to reproducible, safe, and high-precision surgery as it may then be performed at acceptable costs and effort. This is because image-based techniques avoid the need of specialized equipment and seamlessly integrate with contemporary workflows. Furthermore, it is expected that image-based navigation techniques will play a major role in enabling mixed reality environments, as well as autonomous and robot-assisted workflows. A critical component of image guidance is 2D/3D registration, a technique to estimate the spatial relationships between 3D structures, e.g., preoperative volumetric imagery or models of surgical instruments, and 2D images thereof, such as intraoperative X-ray fluoroscopy or endoscopy. While image-based 2D/3D registration is a mature technique, its transition from the bench to the bedside has been restrained by well-known challenges, including brittleness with respect to optimization objective, hyperparameter selection, and initialization, difficulties in dealing with inconsistencies or multiple objects, and limited single-view performance. One reason these challenges persist today is that analytical solutions are likely inadequate considering the complexity, variability, and high-dimensionality of generic 2D/3D registration problems. The recent advent of machine learning-based approaches to imaging problems that, rather than specifying the desired functional mapping, approximate it using highly expressive parametric models holds promise for solving some of the notorious challenges in 2D/3D registration. In this manuscript, we review the impact of machine learning on 2D/3D registration to systematically summarize the recent advances made by introduction of this novel technology. Grounded in these insights, we then offer our perspective on the most pressing needs, significant open problems, and possible next steps.

[1]  Jiebo Luo,et al.  Multiview 2D/3D Rigid Registration via a Point-Of-Interest Network for Tracking and Triangulation , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  V. Lepetit,et al.  EPnP: An Accurate O(n) Solution to the PnP Problem , 2009, International Journal of Computer Vision.

[3]  Sarang C. Joshi,et al.  Real-Time 2D-3D Deformable Registration with Deep Learning and Application to Lung Radiotherapy Targeting , 2019, IPMI.

[4]  Andreas K. Maier,et al.  Learning-Based Correspondence Estimation for 2-D/3-D Registration , 2020, Bildverarbeitung für die Medizin.

[5]  Jon Clarke,et al.  Computer Assisted Knee Replacement Surgery: Is the Movement Mainstream? , 2014 .

[6]  William M. Wells,et al.  Bayesian characterization of uncertainty in intra-subject non-rigid registration , 2013, Medical Image Anal..

[7]  Stephen M. Pizer,et al.  Local Regression Learning via Forest Classification for 2D/3D Deformable Registration , 2013, MCV.

[8]  J. Siewerdsen,et al.  3D–2D registration for surgical guidance: effect of projection view angles on registration accuracy , 2014, Physics in medicine and biology.

[9]  Danni Ai,et al.  Iterative closest graph matching for non-rigid 3D/2D coronary arteries registration , 2020, Comput. Methods Programs Biomed..

[10]  Rui Liao,et al.  Constrained Registration for Motion Compensation in Atrial Fibrillation Ablation Procedures , 2012, IEEE Transactions on Medical Imaging.

[11]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[12]  Josien P. W. Pluim,et al.  The truth is hard to make: Validation of medical image registration , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[13]  Elsc_Admin.  Current state of computer navigation and robotics in unicompartmental and total knee arthroplasty : a systematic review with meta-analysis , 2016 .

[14]  Leo Joskowicz,et al.  Current state of computer navigation and robotics in unicompartmental and total knee arthroplasty: a systematic review with meta-analysis , 2016, Knee Surgery, Sports Traumatology, Arthroscopy.

[15]  Guy Marchal,et al.  Multimodality image registration by maximization of mutual information , 1997, IEEE Transactions on Medical Imaging.

[16]  Amir Alansary,et al.  Predicting Slice-to-Volume Transformation in Presence of Arbitrary Subject Motion , 2017, MICCAI.

[17]  Tianmiao Wang,et al.  Transfer Learning for Nonrigid 2D/3D Cardiovascular Images Registration , 2020, IEEE Journal of Biomedical and Health Informatics.

[18]  Russell H. Taylor,et al.  Image-based navigation for functional endoscopic sinus surgery using structure from motion , 2016, SPIE Medical Imaging.

[19]  Yoshinobu Sato,et al.  Robust patella motion tracking using intensity-based 2D-3D registration on dynamic bi-plane fluoroscopy: towards quantitative assessment in MPFL reconstruction surgery , 2016, SPIE Medical Imaging.

[20]  Boštjan Likar,et al.  Simultaneous 3D-2D image registration and C-arm calibration: Application to endovascular image-guided interventions. , 2015, Medical physics.

[21]  Yoshinobu Sato,et al.  Recovery of 3D rib motion from dynamic chest radiography and CT data using local contrast normalization and articular motion model , 2019, Medical Image Anal..

[22]  U. Mezger,et al.  Navigation in surgery , 2013, Langenbeck's Archives of Surgery.

[23]  Yoshinobu Sato,et al.  Automated CT Segmentation of Diseased Hip Using Hierarchical and Conditional Statistical Shape Models , 2013, MICCAI.

[24]  Fabien Scalzo,et al.  Similarity Metric Learning for 2D to 3D Registration of Brain Vasculature , 2016, ISVC.

[25]  Andreas Maier,et al.  Metric-Driven Learning of Correspondence Weighting for 2-D/3-D Image Registration , 2018, GCPR.

[26]  Z. Jane Wang,et al.  Real-time 2D/3D registration via CNN regression , 2015, 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI).

[27]  Jing Wu,et al.  Fully automatic initialization of two-dimensional–three-dimensional medical image registration using hybrid classifier , 2015, Journal of medical imaging.

[28]  Russell H. Taylor,et al.  Endoscopic navigation in the clinic: registration in the absence of preoperative imaging , 2019, International Journal of Computer Assisted Radiology and Surgery.

[29]  Bishesh Khanal,et al.  Computing CNN Loss and Gradients for Pose Estimation with Riemannian Geometry , 2018, MICCAI.

[30]  Regina Pohle-Fröhlich,et al.  Deep similarity learning using a Siamese ResNet trained on similarity labels from disparity maps of cerebral MRA MIP pairs , 2020, Medical Imaging: Image Processing.

[31]  R Fahrig,et al.  Marker-free motion correction in weight-bearing cone-beam CT of the knee joint. , 2016, Medical physics.

[32]  Ding-Xuan Zhou,et al.  Universality of Deep Convolutional Neural Networks , 2018, Applied and Computational Harmonic Analysis.

[33]  Jeffrey H. Siewerdsen,et al.  Data-driven detection and registration of spine surgery instrumentation in intraoperative images , 2020, Medical Imaging: Image-Guided Procedures.

[34]  Rui Liao,et al.  Dilated FCN for Multi-Agent 2D/3D Medical Image Registration , 2017, AAAI.

[35]  Bostjan Likar,et al.  A review of 3D/2D registration methods for image-guided interventions , 2012, Medical Image Anal..

[36]  Uros Mitrovic,et al.  Automatic Detection of Misalignment in Rigid 3D-2D Registration , 2013, CLIP.

[37]  M. Weigl,et al.  Technical and Nontechnical Skills in Surgery: A Simulated Operating Room Environment Study. , 2019, Spine.

[38]  Peter Kazanzides,et al.  Cross-modal self-supervised representation learning for gesture and skill recognition in robotic surgery , 2021, International Journal of Computer Assisted Radiology and Surgery.

[39]  Lilian Calvet,et al.  Detecting the occluding contours of the uterus to automatise augmented laparoscopy: score, loss, dataset, evaluation and user study , 2020, International Journal of Computer Assisted Radiology and Surgery.

[40]  Pingkun Yan,et al.  Deep learning in medical image registration: a survey , 2020, Machine Vision and Applications.

[41]  Cong Gao,et al.  Fiducial-free 2D/3D registration of the proximal femur for robot-assisted femoroplasty , 2020, Medical Imaging: Image-Guided Procedures.

[42]  T. Lund,et al.  A new approach to computer-aided spine surgery: fluoroscopy-based surgical navigation , 2000, European Spine Journal.

[43]  Mathias Unberath,et al.  CAI4CAI: The Rise of Contextual Artificial Intelligence in Computer-Assisted Interventions , 2019, Proceedings of the IEEE.

[44]  Yingying Guo,et al.  Surgical Navigation in Orthopedics: Workflow and System Review. , 2018, Advances in experimental medicine and biology.

[45]  N. Hafezi-Nejad,et al.  Vertebroplasty and kyphoplasty in the USA from 2004 to 2017: national inpatient trends, regional variations, associated diagnoses, and outcomes , 2020, Journal of NeuroInterventional Surgery.

[46]  Nassir Navab,et al.  Learning to detect anatomical landmarks of the pelvis in X-rays from arbitrary views , 2019, International Journal of Computer Assisted Radiology and Surgery.

[47]  Heinz Handels,et al.  A multilevel Markov Chain Monte Carlo approach for uncertainty quantification in deformable registration , 2018, Medical Imaging.

[48]  Jian Wang,et al.  Learning an Attention Model for Robust 2-D/3-D Registration Using Point-To-Plane Correspondences , 2020, IEEE Transactions on Medical Imaging.

[49]  Hongbin Zha,et al.  Temporal Consistent 2D-3D Registration of Lateral Cephalograms and Cone-Beam Computed Tomography Images , 2018, MLMI@MICCAI.

[50]  W. Hsu,et al.  Effectiveness of Bioskills Training in Spinal Surgery , 2021, Contemporary Spine Surgery.

[51]  Z. Jane Wang,et al.  A CNN Regression Approach for Real-Time 2D/3D Registration , 2016, IEEE Transactions on Medical Imaging.

[52]  Tianmiao Wang,et al.  Transfer Learning for Rigid 2D/3D Cardiovascular Images Registration , 2019, PRCV.

[53]  Ying Sun,et al.  A Review of Recent Advances in Registration Techniques Applied to Minimally Invasive Therapy , 2013, IEEE Transactions on Multimedia.

[54]  Xiang Chen,et al.  Deep learning in medical image registration , 2020, Progress in Biomedical Engineering.

[55]  Hongbin Zha,et al.  Non-rigid Craniofacial 2D-3D Registration Using CNN-Based Regression , 2017, DLMIA/ML-CDS@MICCAI.

[56]  Caroline Petitjean,et al.  A review of 3 D / 2 D registration methods for image-guided interventions , 2016 .

[57]  A Uneri,et al.  Intraoperative evaluation of device placement in spine surgery using known-component 3D–2D image registration , 2017, Physics in medicine and biology.

[58]  Nassir Navab,et al.  Towards clinical translation of augmented orthopedic surgery: from pre-op CT to intra-op x-ray via RGBD sensing , 2018, Medical Imaging.

[59]  Petros Koumoutsakos,et al.  Reducing the Time Complexity of the Derandomized Evolution Strategy with Covariance Matrix Adaptation (CMA-ES) , 2003, Evolutionary Computation.

[60]  Stefan Klein,et al.  Comparative Evaluation of Regression Methods for 3D-2D Image Registration , 2012, ICANN.

[61]  Antonio Pepe,et al.  Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality , 2020, ML-CDS/CLIP@MICCAI.

[62]  M. Powell.  The BOBYQA algorithm for bound constrained optimization without derivatives , 2009 .

[63]  Stephen M. Pizer,et al.  2D/3D image registration using regression learning , 2013, Comput. Vis. Image Underst..

[64]  S. Speidel,et al.  Machine Learning for Surgical Phase Recognition: A Systematic Review. , 2020, Annals of surgery.

[65]  Russell H. Taylor,et al.  Localizing dexterous surgical tools in X-ray for image-based navigation , 2019, ArXiv.

[66]  Mathias Unberath,et al.  UI-Net: Interactive Artificial Neural Networks for Iterative Image Segmentation Based on a User Model , 2017, VCBM.

[67]  Xióngbiao Luó,et al.  Towards Multiple Instance Learning and Hermann Weyl's Discrepancy for Robust Image-Guided Bronchoscopic Intervention , 2019, MICCAI.

[68]  Rui Liao,et al.  Pairwise domain adaptation module for CNN-based 2-D/3-D registration , 2018, Journal of medical imaging.

[69]  Nassir Navab,et al.  DeepDRR - A Catalyst for Machine Learning in Fluoroscopy-guided Procedures , 2018, MICCAI.

[70]  Graeme P. Penney,et al.  Fully automated 2D-3D registration and verification , 2015, Medical Image Anal..

[71]  Li Chen,et al.  A Novel Neurosurgery Registration Pipeline Based on Heat Maps and Anatomic Facial Feature Points , 2019, 2019 12th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).

[72]  Yoshinobu Sato,et al.  Analysis of forearm rotational motion using biplane fluoroscopic intensity-based 2D-3D matching. , 2019, Journal of biomechanics.

[73]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement , 2009, BMJ.

[74]  Marc Niethammer,et al.  Quicksilver: Fast predictive image registration – A deep learning approach , 2017, NeuroImage.

[75]  Stephen M. Pizer,et al.  Real-Time 2D/3D Deformable Registration Using Metric Learning , 2012, MCV.

[76]  Mathias Unberath,et al.  Towards Fully Automatic X-Ray to CT Registration , 2019, MICCAI.

[77]  D. Hawkes,et al.  2D-3D Registration Accuracy Estimation for Optimised Planning of Image-Guided Pancreatobiliary Interventions , 2016, MICCAI.

[78]  Marc Modat,et al.  Label-driven weakly-supervised learning for multimodal deformarle image registration , 2017, 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018).

[79]  Jim Graham,et al.  Automatic Inference and Measurement of 3D Carpal Bone Kinematics From Single View Fluoroscopic Sequences , 2013, IEEE Transactions on Medical Imaging.

[80]  Liang Zhao,et al.  Multi-View Point-Based Registration for Native Knee Kinematics Measurement with Feature Transfer Learning , 2020 .

[81]  M. Figl,et al.  2D/3D registration of endoscopic ultrasound to CT volume data , 2008, Physics in medicine and biology.

[82]  Russell H. Taylor,et al.  Pose Estimation of Periacetabular Osteotomy Fragments With Intraoperative X-Ray Navigation , 2019, IEEE Transactions on Biomedical Engineering.

[83]  Yabo Fu,et al.  Deep Learning in Medical Image Registration: A Review , 2020, Physics in medicine and biology.

[84]  G Kleinszig,et al.  Fracture reduction planning and guidance in orthopaedic trauma surgery via multi-body image registration , 2020, Medical Image Anal..

[85]  Kawal S. Rhode,et al.  3D/2D model-to-image registration by imitation learning for cardiac procedures , 2018, International Journal of Computer Assisted Radiology and Surgery.

[86]  Cong Gao,et al.  Extended Capture Range of Rigid 2D/3D Registration by Estimating Riemannian Pose Gradients , 2020, MLMI@MICCAI.

[87]  Jeffrey H Siewerdsen,et al.  Robotic drill guide positioning using known-component 3D–2D image registration , 2018, Journal of medical imaging.

[88]  Cong Gao,et al.  Learning to Avoid Poor Images: Towards Task-aware C-arm Cone-beam CT Trajectories , 2019, MICCAI.

[89]  Yi Xie,et al.  Single shot 2D3D image regisraton , 2017, 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI).

[90]  Leo Joskowicz,et al.  Computer Aided Orthopaedic Surgery: Incremental shift or paradigm change? , 2016, Medical Image Anal..

[91]  A. Hodgson,et al.  Deep learning‐based X‐ray inpainting for improving spinal 2D‐3D registration , 2021, The international journal of medical robotics + computer assisted surgery : MRCAS.

[92]  Cong Gao,et al.  Generalizing Spatial Transformers to Projective Geometry with Applications to 2D/3D Registration , 2020, MICCAI.

[93]  Russell H. Taylor,et al.  Fiducial-Free 2D/3D Registration for Robot-Assisted Femoroplasty , 2020, IEEE Transactions on Medical Robotics and Bionics.

[94]  Mert R. Sabuncu,et al.  Unsupervised Learning of Probabilistic Diffeomorphic Registration for Images and Surfaces , 2019, Medical Image Anal..

[95]  Jorge Nocedal,et al.  On the limited memory BFGS method for large scale optimization , 1989, Math. Program..

[96]  Iulian Iordachita,et al.  SCADE: Simultaneous Sensor Calibration and Deformation Estimation of FBG-Equipped Unmodeled Continuum Manipulators , 2020, IEEE Transactions on Robotics.

[97]  Hervé Delingette,et al.  Quantifying Registration Uncertainty With Sparse Bayesian Modelling , 2017, IEEE Transactions on Medical Imaging.

[98]  Nassir Navab,et al.  X-ray-transform Invariant Anatomical Landmark Detection for Pelvic Trauma Surgery , 2018, MICCAI.

[99]  T van Walsum,et al.  Respiratory motion estimation in x-ray angiography for improved guidance during coronary interventions , 2015, Physics in medicine and biology.

[100]  Cong Gao,et al.  Automatic annotation of hip anatomy in fluoroscopy for robust and efficient 2D/3D registration , 2019, International Journal of Computer Assisted Radiology and Surgery.

[101]  Qingyu Zhao,et al.  Local Metric Learning in 2D/3D Deformable Registration With Application in the Abdomen , 2014, IEEE Transactions on Medical Imaging.

[102]  Nassir Navab,et al.  Enabling machine learning in X-ray-based procedures via realistic simulation of image formation , 2019, International Journal of Computer Assisted Radiology and Surgery.

[103]  Yoshito Otake,et al.  Robust 3D–2D image registration: application to spine interventions and vertebral labeling in the presence of anatomical deformation , 2013, Physics in medicine and biology.

[104]  Mathias Unberath,et al.  An Interpretable Approach to Automated Severity Scoring in Pelvic Trauma , 2021, MICCAI.

[105]  Cong Gao,et al.  A learning-based method for online adjustment of C-arm Cone-beam CT source trajectories for artifact avoidance , 2020, International Journal of Computer Assisted Radiology and Surgery.

[106]  Hongbin Zha,et al.  Non-Rigid 2D-3D Registration Using Convolutional Autoencoders , 2020, 2020 IEEE 17th International Symposium on Biomedical Imaging (ISBI).

[107]  Markus Kowarschik,et al.  Deep Learning compatible Differentiable X-ray Projections for Inverse Rendering , 2021, Bildverarbeitung für die Medizin.

[108]  Gregory Hager,et al.  Vision-based navigation in image-guided interventions. , 2011, Annual review of biomedical engineering.

[109]  J. Birkmeyer,et al.  Surgical skill and complication rates after bariatric surgery. , 2013, The New England journal of medicine.

[110]  N. Sugano.  Computer-assisted orthopedic surgery , 2003, Journal of orthopaedic science : official journal of the Japanese Orthopaedic Association.

[111]  Xia Hu,et al.  Techniques for interpretable machine learning , 2018, Commun. ACM.