Multi-Modality Imaging: A Software Fusion and Image-Guided Therapy Perspective

With the introduction of computers in medical imaging, which were popularized with the presentation of Hounsfield’s ground-breaking work in 1971, numerical image reconstruction and analysis of medical images became a vital part of medical imaging research. While mathematical aspects of reconstruction dominated research in the beginning, a growing body of literature attests to the progress made over the past 30 years in image fusion. This article describes the historical development of non-deformable software-based image co-registration and it’s role in the context of hybrid imaging and provides an outlook on future developments.

[1]  Michael A. Speidel,et al.  Real-time pose estimation of devices from x-ray images: Application to x-ray/echo registration for cardiac interventions , 2016, Medical Image Anal..

[2]  Chia-Ling Tsai,et al.  The Edge-Driven Dual-Bootstrap Iterative Closest Point Algorithm for Registration of Multimodal Fluorescein Angiogram Sequence , 2010, IEEE Transactions on Medical Imaging.

[3]  B. J. Perry,et al.  Computerized transverse axial scanning (tomography). 3. Radiation dose considerations. , 1973, The British journal of radiology.

[4]  Benjamin Berkels,et al.  Survey of Non-Rigid Registration Tools in Medicine , 2017, Journal of Digital Imaging.

[5]  D. R. Fish,et al.  A patient-to-computed-tomography image registration method based on digitally reconstructed radiographs. , 1994, Medical physics.

[6]  Terry M. Peters,et al.  Image registration of ex-vivo MRI to sparsely sectioned histology of hippocampal and neocortical temporal lobe specimens , 2013, NeuroImage.

[7]  Wolfgang Birkfellner,et al.  Applied Medical Image Processing: A Basic Course , 2010 .

[8]  Wolfgang Birkfellner,et al.  Registration of DRRs and portal images for verification of stereotactic body radiotherapy: a feasibility study in lung cancer treatment. , 2007, Physics in medicine and biology.

[9]  Robert J. Maciunas,et al.  Registration of head volume images using implantable fiducial markers , 1997, IEEE Transactions on Medical Imaging.

[10]  C R Meyer,et al.  Motion correction in fMRI via registration of individual slices into an anatomical volume , 1999, Magnetic resonance in medicine.

[11]  W Birkfellner,et al.  Automatic quantification of multi-modal rigid registration accuracy using feature detectors. , 2016, Physics in medicine and biology.

[12]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[13]  C J Taylor,et al.  Anatomical statistical models and their role in feature extraction. , 2004, The British journal of radiology.

[14]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[15]  Jing Ren,et al.  Dynamic 2D Ultrasound and 3D CT Image Registration of the Beating Heart , 2009, IEEE Transactions on Medical Imaging.

[16]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[17]  D E Kuhl,et al.  Transmission scanning: a useful adjunct to conventional emission scanning for accurately keying isotope deposition to radiographic anatomy. , 1966, Radiology.

[18]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[19]  D. Jones,et al.  A frameless method for stereotactic radiotherapy. , 1993, The British journal of radiology.

[20]  R. Ledley,et al.  Computerized Transaxial X-ray Tomography of the Human Body , 1974, Science.

[21]  Gregory C Sharp,et al.  Scale invariant feature transform in adaptive radiation therapy: a tool for deformable image registration assessment and re-planning indication , 2013, Physics in medicine and biology.

[22]  M N Maisey,et al.  Computer-aided interpretation of SPECT images of the brain using an MRI-derived 3D neuro-anatomical atlas. , 1991, Medical informatics = Medecine et informatique.

[23]  Luc Van Gool,et al.  Speeded-Up Robust Features (SURF) , 2008, Comput. Vis. Image Underst..

[24]  Bostjan Likar,et al.  A review of 3D/2D registration methods for image-guided interventions , 2012, Medical Image Anal..

[25]  Guy Marchal,et al.  Multimodality image registration by maximization of mutual information , 1997, IEEE Transactions on Medical Imaging.

[26]  Jonathan S. Lewin,et al.  Slice-to-volume registration and its potential application to interventional MRI-guided radio-frequency thermal ablation of prostate cancer , 2003, IEEE Transactions on Medical Imaging.

[27]  D. Peck Digital Imaging and Communications in Medicine (DICOM): A Practical Introduction and Survival Guide , 2009, Journal of Nuclear Medicine.

[28]  J M Galvin,et al.  The use of digitally reconstructed radiographs for three-dimensional treatment planning and CT-simulation. , 1995, International journal of radiation oncology, biology, physics.

[29]  Zuofeng Li,et al.  The selection of portal aperture using interactively displayed eye sections , 1992 .

[30]  Marco Riboldi,et al.  A Hybrid Image Registration and Matching Framework for Real-Time Motion Tracking in MRI-Guided Radiotherapy , 2018, IEEE Transactions on Biomedical Engineering.

[31]  Nataliya Kovalchuk,et al.  Radiotherapy planning. , 2015, PET clinics.

[32]  Ge Wang,et al.  Vision 20/20: Simultaneous CT-MRI--Next chapter of multimodality imaging. , 2015, Medical physics.

[33]  H Shirato,et al.  Use of an implanted marker and real-time tracking of the marker for the positioning of prostate and bladder cancers. , 2000, International journal of radiation oncology, biology, physics.

[34]  David J. Hawkes,et al.  Alignment of sparse freehand 3-D ultrasound with preoperative images of the liver using models of respiratory motion and deformation , 2005, IEEE Transactions on Medical Imaging.

[35]  K S Lam,et al.  An on-line electronic portal imaging system for external beam radiotherapy. , 1986, The British journal of radiology.

[36]  G. Hounsfield Computerized transverse axial scanning (tomography): Part I. Description of system. 1973. , 1973, The British journal of radiology.

[37]  Paul J. Besl,et al.  Method for registration of 3-D shapes , 1992, Other Conferences.

[38]  Gabor Fichtinger,et al.  Monitoring tumor motion by real time 2D/3D registration during radiotherapy , 2012, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[39]  Nikos Paragios,et al.  Slice-to-volume deformable registration: efficient one-shot consensus between plane selection and in-plane deformation , 2015, International Journal of Computer Assisted Radiology and Surgery.

[40]  Li Bai,et al.  Automatic Best Reference Slice Selection for Smooth Volume Reconstruction of a Mouse Brain From Histological Images , 2010, IEEE Transactions on Medical Imaging.

[41]  Jay B. West,et al.  The distribution of target registration error in rigid-body point-based registration , 2001, IEEE Transactions on Medical Imaging.

[42]  H Bergmann,et al.  In-vitro assessment of a registration protocol for image guided implant dentistry. , 2001, Clinical oral implants research.

[43]  B. Ardekani,et al.  A Fully Automatic Multimodality Image Registration Algorithm , 1995, Journal of computer assisted tomography.

[44]  J. Kettenbach,et al.  Stochastic rank correlation: a robust merit function for 2D/3D registration of image data obtained at different energies. , 2009, Medical physics.

[45]  Uwe Pietrzyk,et al.  Does PET/MR in human brain imaging provide optimal co-registration? A critical reflection , 2012, Magnetic Resonance Materials in Physics, Biology and Medicine.

[46]  Katia Parodi,et al.  Investigating CT to CBCT image registration for head and neck proton therapy as a tool for daily dose recalculation. , 2015, Medical physics.

[47]  E. Hoffman,et al.  A positron-emission transaxial tomograph for nuclear imaging (PETT). , 1975, Radiology.

[48]  David G. Lowe,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004, International Journal of Computer Vision.

[49]  Bostjan Likar,et al.  Validation for 2D/3D registration. II: The comparison of intensity- and gradient-based merit functions using a new gold standard data set. , 2011, Medical physics.

[50]  Jakob Spork,et al.  High-performance GPU based Rendering for Real-Time, rigid 2D/3D-Image Registration in Radiation Oncology , 2011 .

[51]  M. Figl,et al.  2D/3D registration of endoscopic ultrasound to CT volume data , 2008, Physics in medicine and biology.

[52]  Thomas Guerrero,et al.  A reference dataset for deformable image registration spatial accuracy evaluation using the COPDgene study archive , 2013, Physics in medicine and biology.

[53]  Bostjan Likar,et al.  A framework for automatic creation of gold-standard rigid 3D–2D registration datasets , 2017, International Journal of Computer Assisted Radiology and Surgery.

[54]  P C Lauterbur,et al.  Progress in n.m.r. zeugmatography imaging. , 1980, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.

[55]  M W Vannier,et al.  Image-based dose planning of intracavitary brachytherapy: registration of serial-imaging studies using deformable anatomic templates. , 2001, International journal of radiation oncology, biology, physics.

[56]  W. Birkfellner,et al.  Clinical Assessment of 2D/3D Registration Accuracy in 4 Major Anatomic Sites Using On-Board 2D Kilovoltage Images for 6D Patient Setup , 2015, Technology in cancer research & treatment.

[57]  Wolfgang Birkfellner,et al.  Fast DRR splat rendering using common consumer graphics hardware. , 2007, Medical physics.

[58]  Nikos Paragios,et al.  Slice-to-volume medical image registration: a survey , 2017, Medical Image Anal..

[59]  W Birkfellner,et al.  A navigation system for flexible endoscopes using abdominal 3D ultrasound , 2014, Physics in medicine and biology.

[60]  Max A. Viergever,et al.  Normalized mutual information based registration using k-means clustering and shading correction , 2006, Medical Image Anal..

[61]  Graeme P. Penney,et al.  Standardized evaluation methodology for 2-D-3-D registration , 2005, IEEE Transactions on Medical Imaging.

[62]  Logan W. Clements,et al.  Robust surface registration using salient anatomical features for image-guided liver surgery: algorithm and validation. , 2008, Medical physics.

[63]  Wolfgang Birkfellner,et al.  Real-time 2D/3D registration using kV-MV image pairs for tumor motion tracking in image guided radiotherapy , 2013, Acta oncologica.

[64]  M. Figl,et al.  Validation for 2D/3D registration. I: A new gold standard data set. , 2011, Medical physics.

[65]  E. Samsøe,et al.  Dosimetric and geometric evaluation of the use of deformable image registration in adaptive intensity-modulated radiotherapy for head-and-neck cancer , 2014, Journal of radiation research.

[66]  Hans-Peter Meinzer,et al.  Statistical shape models for 3D medical image segmentation: A review , 2009, Medical Image Anal..

[67]  Rafael C. González,et al.  Local Determination of a Moving Contrast Edge , 1985, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Lefteris Livieratos,et al.  A dual modality approach to quantitative quality control in emission tomography , 2005, Physics in medicine and biology.

[69]  Benoit M. Dawant,et al.  Registration of 3-D images using weighted geometrical features , 1996, IEEE Trans. Medical Imaging.

[70]  U Pietrzyk,et al.  Clinical applications of registration and fusion of multimodality brain images from PET, SPECT, CT, and MRI. , 1996, European journal of radiology.

[71]  C. Morandi,et al.  Registration of Translated and Rotated Images Using Finite Fourier Transforms , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[72]  D. Dormont,et al.  Registration in neurosurgery and neuroradiotherapy applications. , 1995, Journal of image guided surgery.

[73]  D. Ruan,et al.  A continuous surface reconstruction method on point cloud captured from a 3D surface photogrammetry system. , 2015, Medical physics.

[74]  S J Zinreich,et al.  In vivo accuracy testing and clinical experience with the ISG Viewing Wand. , 1996, Neurosurgery.

[75]  Jiaolong Yang,et al.  Go-ICP: A Globally Optimal Solution to 3D ICP Point-Set Registration , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[76]  Richard J. Evans,et al.  Development and preliminary evaluation of VISLAN, a surgical planning and guidance system using intra-operative video imaging , 1996, Medical Image Anal..

[77]  T. Turkington,et al.  Accuracy of registration of PET, SPECT and MR images of a brain phantom. , 1993, Journal of nuclear medicine : official publication, Society of Nuclear Medicine.

[78]  Richard O. Duda,et al.  Pattern classification and scene analysis , 1974, A Wiley-Interscience publication.

[79]  H. Anger,et al.  MAPPING THE DISTRIBUTION OF GAMMA-RAY-EMITTING ISOTOPES WITH THE SCINTILLATION CAMERA , 1958 .

[80]  Timothy F. Cootes,et al.  A Unified Information-Theoretic Approach to Groupwise Non-rigid Registration and Model Building , 2005, IPMI.

[81]  Lena Maier-Hein,et al.  Convergent Iterative Closest-Point Algorithm to Accomodate Anisotropic and Inhomogenous Localization Error , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[82]  Wolfgang Birkfellner,et al.  Automatic patient alignment system using 3D ultrasound. , 2013, Medical physics.

[83]  M Holman,et al.  The selection of portal aperture using interactively displayed Beam's Eye Sections. , 1992, International journal of radiation oncology, biology, physics.

[84]  P Hofman,et al.  Detection of fiducial gold markers for automatic on-line megavoltage position verification using a marker extraction kernel (MEK). , 2000, International journal of radiation oncology, biology, physics.

[85]  Syed Muhammad Anwar,et al.  Deep Learning in Medical Image Analysis , 2017 .

[86]  Jay B. West,et al.  Predicting error in rigid-body point-based registration , 1998, IEEE Transactions on Medical Imaging.

[87]  M van Herk,et al.  First clinical experience with a newly developed electronic portal imaging device. , 1990, International journal of radiation oncology, biology, physics.

[88]  Jürgen Weese,et al.  A comparison of similarity measures for use in 2-D-3-D medical image registration , 1998, IEEE Transactions on Medical Imaging.

[89]  William Wells,et al.  Multimodal image registration for preoperative planning and image-guided neurosurgical procedures. , 2011, Neurosurgery clinics of North America.

[90]  J. Kettenbach,et al.  A comparative study on manual and automatic slice-to-volume registration of CT images , 2009, European Radiology.

[91]  J. Ambrose,et al.  Computerized transverse axial scanning (tomography): Part 2. Clinical application* , 1973 .

[92]  M van Herk,et al.  Automatic on-line inspection of patient setup in radiation therapy using digital portal images. , 1993, Medical physics.

[93]  Shinichi Shimizu,et al.  Organ motion in image-guided radiotherapy: lessons from real-time tumor-tracking radiotherapy , 2007, International Journal of Clinical Oncology.

[94]  J. Ambrose,et al.  Computerized transverse axial scanning (tomography). 2. Clinical application. , 1973, The British journal of radiology.

[95]  R. Damadian Tumor Detection by Nuclear Magnetic Resonance , 1971, Science.

[96]  Klaus Radermacher,et al.  A-mode ultrasound-based intra-femoral bone cement detection and 3D reconstruction in RTHR , 2007, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[97]  C. Pelizzari,et al.  Retrospective geometric correlation of MR, CT, and PET images. , 1988, Radiology.

[98]  B. K. Swann,et al.  PET/MR images acquired with a compact MR-compatible PET detector in a 7-T magnet. , 2007, Radiology.

[99]  J. Mazziotta,et al.  MRI‐PET Registration with Automated Algorithm , 1993, Journal of computer assisted tomography.

[100]  Wolfgang Birkfellner,et al.  A quantitative comparison of the performance of three deformable registration algorithms in radiotherapy , 2013, Zeitschrift fur medizinische Physik.

[101]  G. Marchal,et al.  Multi-modal volume registration by maximization of mutual information , 1997 .

[102]  H Bergmann,et al.  Efficient implementation of the rank correlation merit function for 2D/3D registration. , 2010, Physics in medicine and biology.

[103]  P. K. Banerjee,et al.  Registration revisited , 1993, Journal of Neuroscience Methods.

[104]  J. Kettenbach,et al.  Rigid 2D/3D slice-to-volume registration and its application on fluoroscopic CT images. , 2006, Medical physics.

[105]  V.R.S Mani,et al.  Survey of Medical Image Registration , 2013 .

[106]  D L Hill,et al.  Automated three-dimensional registration of magnetic resonance and positron emission tomography brain images by multiresolution optimization of voxel similarity measures. , 1997, Medical physics.

[107]  Wolfgang Birkfellner,et al.  Visualization of Deformable Image Registration Quality Using Local Image Dissimilarity , 2016, IEEE Transactions on Medical Imaging.

[108]  M. Vogele,et al.  Vogele-Bale-Hohner mouthpiece: registration device for frameless stereotactic surgery. , 1998, Radiology.

[109]  Jan J W Lagendijk,et al.  MRI/linac integration. , 2008, Radiotherapy and oncology : journal of the European Society for Therapeutic Radiology and Oncology.

[110]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[111]  P. Mansfield,et al.  Line scan proton spin imaging in biological structures by NMR. , 1976, Physics in medicine and biology.

[112]  Kawal S. Rhode,et al.  Intensity-based 2-D - 3-D registration of cerebral angiograms , 2003, IEEE Transactions on Medical Imaging.

[113]  Marcel van Herk,et al.  Using a contextualized sensemaking model for interaction design: A case study of tumor contouring , 2017, J. Biomed. Informatics.

[114]  C. Pelizzari,et al.  Accurate Three‐Dimensional Registration of CT, PET, and/or MR Images of the Brain , 1989, Journal of computer assisted tomography.

[115]  G. Hounsfield Computerized transverse axial scanning (tomography). 1. Description of system. , 1973, The British journal of radiology.

[116]  J. Strohbehn,et al.  A frameless stereotaxic integration of computerized tomographic imaging and the operating microscope. , 1986, Journal of neurosurgery.

[117]  Gerald Q. Maguire,et al.  Comparison and evaluation of retrospective intermodality brain image registration techniques. , 1997, Journal of computer assisted tomography.

[118]  Tomaz Slivnik,et al.  3-D/2-D registration of CT and MR to X-ray images , 2003, IEEE Transactions on Medical Imaging.

[119]  S. Nill,et al.  Linac-integrated kV-cone beam CT: technical features and first applications. , 2006, Medical dosimetry : official journal of the American Association of Medical Dosimetrists.

[120]  Nikos Paragios,et al.  Deformable Medical Image Registration: A Survey , 2013, IEEE Transactions on Medical Imaging.

[121]  Stefano Alliney,et al.  Digital Image Registration Using Projections , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[122]  Robert L. Galloway,et al.  Cortical surface registration for image-guided neurosurgery using laser-range scanning , 2003, IEEE Transactions on Medical Imaging.

[123]  K. Brock,et al.  Use of image registration and fusion algorithms and techniques in radiotherapy: Report of the AAPM Radiation Therapy Committee Task Group No. 132 , 2017, Medical physics.

[124]  D. Hill,et al.  Registration of MR and CT images for skull base surgery using point-like anatomical features. , 1991, The British journal of radiology.

[125]  Paul Kinahan,et al.  Attenuation correction for a combined 3D PET/CT scanner. , 1998, Medical physics.