Virtual and Augmented Reality Systems for Renal Interventions: A Systematic Review

Purpose: Many virtual and augmented reality systems have been proposed to support renal interventions. This paper reviews such systems employed in the treatment of renal cell carcinoma and renal stones. Methods: A systematic literature search was performed. Inclusion criteria were virtual and augmented reality systems for radical or partial nephrectomy and renal stone treatment, excluding systems solely developed or evaluated for training purposes. Results: In total, 52 research papers were identified and analyzed. Most of the identified literature (87%) deals with systems for renal cell carcinoma treatment. About 44% of the systems have already been employed in clinical practice, but only 20% in studies with ten or more patients. Main challenges remaining for future research include the consideration of organ movement and deformation, human factor issues, and the conduction of large clinical studies. Conclusion: Augmented and virtual reality systems have the potential to improve safety and outcomes of renal interventions. In the last ten years, many technical advances have led to more sophisticated systems, which are already applied in clinical practice. Further research is required to cope with current limitations of virtual and augmented reality assistance in clinical environments.

[1]  A. Darzi,et al.  Using preoperative imaging for intraoperative guidance: a case of mistaken identity , 2016, The international journal of medical robotics + computer assisted surgery : MRCAS.

[2]  A. Cuschieri,et al.  A Systematic Review of Virtual Reality Simulators for Robot-assisted Surgery. , 2016, European urology.

[3]  Lena Maier-Hein,et al.  Robust augmented reality guidance with fluorescent markers in laparoscopic surgery , 2016, International Journal of Computer Assisted Radiology and Surgery.

[4]  Robert Rohling,et al.  Tracked “Pick-Up” Ultrasound for Robot-Assisted Minimally Invasive Surgery , 2016, IEEE Transactions on Biomedical Engineering.

[5]  M. Yiu,et al.  Partial nephrectomy for T1 renal cancer can achieve an equivalent oncological outcome to radical nephrectomy with better renal preservation: the way to go. , 2015, Hong Kong medical journal = Xianggang yi xue za zhi.

[6]  Ghassan Hamarneh,et al.  Simultaneous Multi-Structure Segmentation and 3D Nonrigid Pose Estimation in Image-Guided Robotic Surgery , 2016, IEEE Transactions on Medical Imaging.

[7]  Ghassan Hamarneh,et al.  Endoscopic scene labelling and augmentation using intraoperative pulsatile motion and colour appearance cues with preoperative anatomical priors , 2016, International Journal of Computer Assisted Radiology and Surgery.

[8]  Robert J. Webster,et al.  A novel method for texture-mapping conoscopic surfaces for minimally invasive image-guided kidney surgery , 2016, International Journal of Computer Assisted Radiology and Surgery.

[9]  Ghassan Hamarneh,et al.  Automatic segmentation of occluded vasculature via pulsatile motion analysis in endoscopic robot-assisted partial nephrectomy video , 2015, Medical Image Anal..

[10]  Guang-Zhong Yang,et al.  Pico Lantern: Surface reconstruction and augmented reality in laparoscopic surgery using a pick-up laser projector , 2015, Medical Image Anal..

[11]  Guang-Zhong Yang,et al.  Robust ultrasound probe tracking: initial clinical experiences during robot-assisted partial nephrectomy , 2015, International Journal of Computer Assisted Radiology and Surgery.

[12]  Haruo Takemura,et al.  Stiffness matrix representation of hyper-elasticity for surgical simulation and navigation , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[13]  M. Yao,et al.  Validation of a patient‐specific simulator for laparoscopic renal surgery , 2015, International journal of urology.

[14]  Satoru Muto,et al.  Feasibility and accuracy of computational robot‐assisted partial nephrectomy planning by virtual partial nephrectomy analysis , 2015, International journal of urology : official journal of the Japanese Urological Association.

[15]  Ara Darzi,et al.  The current and future use of imaging in urological robotic surgery: a survey of the European Association of Robotic Urological Surgeons , 2015, The international journal of medical robotics + computer assisted surgery : MRCAS.

[16]  Beat P. Müller-Stich,et al.  Computer-assisted abdominal surgery: new technologies , 2015, Langenbeck's Archives of Surgery.

[17]  Bin Zhang,et al.  Preoperative planning and real-time assisted navigation by three-dimensional individual digital model in partial nephrectomy with three-dimensional laparoscopic system , 2015, International Journal of Computer Assisted Radiology and Surgery.

[18]  Lorenzo Marconi,et al.  EAU guidelines on renal cell carcinoma: 2014 update. , 2010, European urology.

[19]  A. Jemal,et al.  Cancer statistics, 2015 , 2015, CA: a cancer journal for clinicians.

[20]  Naoki Suzuki,et al.  Clinical application of navigation surgery using augmented reality in the abdominal field , 2015, Surgery Today.

[21]  A. Darzi,et al.  Inattention blindness in surgery , 2015, Surgical Endoscopy.

[22]  P. Dasgupta,et al.  Image guided robotic surgery: current evidence for effectiveness in urology. , 2014, Archivio italiano di urologia, andrologia : organo ufficiale [di] Societa italiana di ecografia urologica e nefrologica.

[23]  Maki Sugimoto,et al.  Console‐integrated real‐time three‐dimensional image overlay navigation for robot‐assisted partial nephrectomy with selective arterial clamping: early single‐centre experience with 17 cases , 2014, The international journal of medical robotics + computer assisted surgery : MRCAS.

[24]  W. Rathmell,et al.  Renal cell carcinoma , 2014, BMJ : British Medical Journal.

[25]  Jeffrey A. Cadeddu,et al.  Toward Long-Term and Accurate Augmented-Reality for Monocular Endoscopic Videos , 2014, IEEE Transactions on Biomedical Engineering.

[26]  Ghassan Hamarneh,et al.  Auto Localization and Segmentation of Occluded Vessels in Robot-Assisted Partial Nephrectomy , 2014, MICCAI.

[27]  Ghassan Hamarneh,et al.  Efficient Multi-organ Segmentation in Multi-view Endoscopic Videos Using Pre-operative Priors , 2014, MICCAI.

[28]  Ghassan Hamarneh,et al.  Biomechanical kidney model for predicting tumor displacement in the presence of external pressure load , 2014, 2014 IEEE 11th International Symposium on Biomedical Imaging (ISBI).

[29]  Ara Darzi,et al.  Image guidance for all--TilePro display of 3-dimensionally reconstructed images in robotic partial nephrectomy. , 2014, Urology.

[30]  H. Yokota,et al.  Prediction of open urinary tract in laparoscopic partial nephrectomy by virtual resection plane visualization , 2014, BMC Urology.

[31]  Lena Maier-Hein,et al.  Comparative Validation of Single-Shot Optical Techniques for Laparoscopic 3-D Surface Reconstruction , 2014, IEEE Transactions on Medical Imaging.

[32]  Uwe Zimmermann,et al.  The contemporary role of ablative treatment approaches in the management of renal cell carcinoma (RCC): focus on radiofrequency ablation (RFA), high-intensity focused ultrasound (HIFU), and cryoablation , 2014, World Journal of Urology.

[33]  J. Stolzenburg,et al.  Laparoscopic radical and partial nephrectomy: The clinical efficacy and acceptance of the techniques , 2014, Urology annals.

[34]  N. Saitō,et al.  A novel 3-dimensional image analysis system for case-specific kidney anatomy and surgical simulation to facilitate clampless partial nephrectomy. , 2014, Urology.

[35]  A. Darzi,et al.  Augmented reality partial nephrectomy: examining the current status and future perspectives. , 2014, Urology.

[36]  Hans-Peter Meinzer,et al.  Surgical navigation in urology: European perspective , 2014, Current opinion in urology.

[37]  Chunxiao Liu,et al.  Surgical planning and manual image fusion based on 3D model facilitate laparoscopic partial nephrectomy for intrarenal tumors , 2014, World Journal of Urology.

[38]  Nima Najmaei,et al.  Image‐guided techniques in renal and hepatic interventions , 2013, The international journal of medical robotics + computer assisted surgery : MRCAS.

[39]  Pengfei Shao,et al.  Three-dimensional renal CT angiography for guiding segmental renal artery clamping during laparoscopic partial nephrectomy. , 2013, Clinical radiology.

[40]  Comparison between laparoscopic and open partial nephrectomy: Surgical, oncologic, and functional outcomes , 2013, The Kaohsiung journal of medical sciences.

[41]  Ghassan Hamarneh,et al.  Uncertainty-Encoded Augmented Reality for Robot-Assisted Partial Nephrectomy: A Phantom Study , 2013, AE-CAI.

[42]  Lena Maier-Hein,et al.  Mobile augmented reality for computer-assisted percutaneous nephrolithotomy , 2013, International Journal of Computer Assisted Radiology and Surgery.

[43]  U. Mezger,et al.  Navigation in surgery , 2013, Langenbeck's Archives of Surgery.

[44]  Gian Luca Mariottini,et al.  A Fast and Accurate Feature-Matching Algorithm for Minimally-Invasive Endoscopic Images , 2013, IEEE Transactions on Medical Imaging.

[45]  Robert Rohling,et al.  Motion of the Kidney Between Preoperative and Intraoperative Positioning , 2013, IEEE Transactions on Biomedical Engineering.

[46]  B. Lane,et al.  10-year oncologic outcomes after laparoscopic and open partial nephrectomy. , 2010, The Journal of urology.

[47]  Chunxiao Liu,et al.  Construction of a three-dimensional model of renal stones: comprehensive planning for percutaneous nephrolithotomy and assistance in surgery , 2013, World Journal of Urology.

[48]  Michael J. Daly,et al.  Surgeons blinded by enhanced navigation: the effect of augmented reality on attention , 2013, Surgical Endoscopy.

[49]  K. Makiyama,et al.  Laparoscopic nephroureterectomy in renal pelvic urothelial carcinoma with situs inversus totalis: preoperative training using a patient-specific simulator. , 2012, Urology.

[50]  Victoria Chernyak,et al.  Virtual surgical planning: a novel aid to robot-assisted laparoscopic partial nephrectomy. , 2012, Journal of endourology.

[51]  Manabu Nagasaka,et al.  Development of a patient‐specific simulator for laparoscopic renal surgery , 2012, International journal of urology : official journal of the Japanese Urological Association.

[52]  Ken Chen,et al.  Augmenting intraoperative ultrasound with preoperative magnetic resonance planning models for percutaneous renal access , 2012, BioMedical Engineering OnLine.

[53]  Michael C. Yip,et al.  Tissue Tracking and Registration for Image-Guided Surgery , 2012, IEEE Transactions on Medical Imaging.

[54]  David C. Miller,et al.  A matched comparison of perioperative outcomes of a single laparoscopic surgeon versus a multisurgeon robot-assisted cohort for partial nephrectomy. , 2012, The Journal of urology.

[55]  Christopher S Saigal,et al.  Prevalence of kidney stones in the United States. , 2012, European urology.

[56]  John L. Gore,et al.  Long-term survival following partial vs radical nephrectomy among older patients with early-stage kidney cancer. , 2012, JAMA.

[57]  H. Meinzer,et al.  iPad-assisted percutaneous access to the kidney using marker-based navigation: initial clinical experience. , 2012, European urology.

[58]  I. Gill,et al.  Current progress on augmented reality visualization in endoscopic surgery , 2012, Current opinion in urology.

[59]  R. Galloway,et al.  Image-guided robotic surgery: update on research and potential applications in urologic surgery , 2012, Current opinion in urology.

[60]  Masahiko Nakamoto,et al.  Three-dimensional reconstruction of renovascular-tumor anatomy to facilitate zero-ischemia partial nephrectomy. , 2012, European urology.

[61]  杨帆 A Shape-Optimized Framework for Kidney Segmentation in Ultrasound Images Using NLTV Denoising and DRLSE , 2012 .

[62]  Jeffrey A. Cadeddu,et al.  Adaptive Multi-Affine (AMA) feature-matching algorithm and its application to Minimally-Invasive Surgery images , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[63]  J. Marescaux,et al.  Augmented reality in laparoscopic surgical oncology. , 2011, Surgical oncology.

[64]  J. Kaouk,et al.  Complications after robotic partial nephrectomy at centers of excellence: multi-institutional analysis of 450 cases. , 2011, The Journal of urology.

[65]  R. Galloway,et al.  Kidney deformation and intraprocedural registration: a study of elements of image-guided kidney surgery. , 2011, Journal of endourology.

[66]  Courtenay Glisson,et al.  The use of virtual fiducials in image-guided kidney surgery , 2011, Medical Imaging.

[67]  R. Rohling,et al.  Registration of CT to 3D ultrasound using near-field fiducial localization: A feasibility study , 2011, Computer aided surgery : official journal of the International Society for Computer Aided Surgery.

[68]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[69]  Salvatore Micali,et al.  New trends in minimally invasive urological surgery. What is beyond the robot? , 2013, World Journal of Urology.

[70]  Jérôme Szewczyk,et al.  An algorithm for calculi segmentation on ureteroscopic images , 2011, International Journal of Computer Assisted Radiology and Surgery.

[71]  Ravindra Sabnis,et al.  Percutaneous renal access training: content validation comparison between a live porcine and a virtual reality (VR) simulation model , 2010, BJU international.

[72]  Chunming Li,et al.  Distance Regularized Level Set Evolution and Its Application to Image Segmentation , 2010, IEEE Transactions on Image Processing.

[73]  M. Reyes,et al.  Computer aided surgery for percutaneous nephrolithotomy: Clinical requirement analysis and system design , 2010, 2010 Annual International Conference of the IEEE Engineering in Medicine and Biology.

[74]  Luc Soler,et al.  Bulk modulus and volume variation measurement of the liver and the kidneys in vivo using abdominal kinetics during free breathing , 2010, Comput. Methods Programs Biomed..

[75]  Terry M. Peters,et al.  Fused Video and Ultrasound Images for Minimally Invasive Partial Nephrectomy: A Phantom Study , 2010, MICCAI.

[76]  Vincent Lepetit,et al.  BRIEF: Binary Robust Independent Elementary Features , 2010, ECCV.

[77]  Kazuhiro Araki,et al.  Surgical navigation using three-dimensional computed tomography images fused intraoperatively with live video. , 2010, Journal of endourology.

[78]  I. Gill,et al.  Clinical practice. Small renal mass. , 2010, The New England journal of medicine.

[79]  F. Kim,et al.  7-year oncological outcomes after laparoscopic and open partial nephrectomy , 2010 .

[80]  A. Scherpbier,et al.  Virtual reality laparoscopic nephrectomy simulator is lacking in construct validity. , 2010, Journal of endourology.

[81]  Minsu Cho,et al.  Feature correspondence and deformable object matching via agglomerative correspondence clustering , 2009, 2009 IEEE 12th International Conference on Computer Vision.

[82]  Ali Serdar Gözen,et al.  Augmented reality: a new tool to improve surgical accuracy during laparoscopic partial nephrectomy? Preliminary in vitro and in vivo results. , 2009, European urology.

[83]  D. Moher,et al.  Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement , 2009, BMJ.

[84]  Carl-Fredrik Westin,et al.  Towards real time 2D to 3D registration for ultrasound-guided endoscopic and laparoscopic procedures , 2009, International Journal of Computer Assisted Radiology and Surgery.

[85]  F. Montorsi,et al.  Complications of laparoscopic surgery for renal masses: prevention, management, and comparison with the open experience. , 2009, European urology.

[86]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[87]  Kazuhiro Araki,et al.  Usefulness of panoramic views for novice surgeons doing retroperitoneal laparoscopic nephrectomy , 2009, International journal of urology : official journal of the Japanese Urological Association.

[88]  Y. Funahashi,et al.  Ischemic renal damage after nephron-sparing surgery in patients with normal contralateral kidney. , 2009, European urology.

[89]  Kurt Konolige,et al.  CenSurE: Center Surround Extremas for Realtime Feature Detection and Matching , 2008, ECCV.

[90]  Logan W. Clements,et al.  Feasibility study for image-guided kidney surgery: assessment of required intraoperative surface for accurate physical to image space registrations. , 2008, Medical physics.

[91]  Beat P. Müller-Stich,et al.  Soft tissue navigation for laparoscopic partial nephrectomy , 2008, International Journal of Computer Assisted Radiology and Surgery.

[92]  I. Gill,et al.  Imaging-assisted endoscopic surgery: Cleveland Clinic experience. , 2008, Journal of endourology.

[93]  S. Duke Herrell,et al.  A kidney deformation model for use in non-rigid registration during image-guided surgery , 2008, SPIE Medical Imaging.

[94]  Jocelyne Troccaz,et al.  POD-02.08: Aid to percutaneous renal access by virtual projection of the ultrasound puncture tract onto fluoroscopic images , 2007 .

[95]  Bernhard Preim,et al.  Visual Computing for Medicine: Theory, Algorithms, and Applications , 2007 .

[96]  Louis R Kavoussi,et al.  Comparison of 1,800 laparoscopic and open partial nephrectomies for single renal tumors. , 2007, The Journal of urology.

[97]  Joshua Stern,et al.  Percutaneous renal access simulators. , 2007, Journal of endourology.

[98]  J. Alison Noble,et al.  Ultrasound image segmentation: a survey , 2006, IEEE Transactions on Medical Imaging.

[99]  Terry M. Peters,et al.  3D segmentation of kidney tumors from freehand 2D ultrasound , 2006, SPIE Medical Imaging.

[100]  Y Payan,et al.  Computer‐assisted access to the kidney , 2005, The international journal of medical robotics + computer assisted surgery : MRCAS.

[101]  D. Gervais,et al.  Radiofrequency ablation of renal cell carcinoma: part 1, Indications, results, and role in patient management over a 6-year period and ablation of 100 tumors. , 2005, AJR. American journal of roentgenology.

[102]  Jun Xie,et al.  Segmentation of kidney from ultrasound images based on texture and shape priors , 2005, IEEE Transactions on Medical Imaging.

[103]  William M. Wells,et al.  Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation , 2004, IEEE Transactions on Medical Imaging.

[104]  M. Marangella,et al.  Epidemiology of nephrolithiasis. , 2000, JN. Journal of Nephrology (Milano. 1992).

[105]  Ronald Azuma,et al.  A Survey of Augmented Reality , 1997, Presence: Teleoperators & Virtual Environments.

[106]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..