Remote surgeon hand motion and occlusion removal in mixed reality in breast surgical telepresence: Rural and remote care

Mixed reality in surgical telepresence has not been practically implemented as yet and research done, until now, is only theoretical. Aiming to implement a proposed new solution by merging virtual and augmented realities and bringing about improvements by removing occlusion and the noise caused by hand motions during surgery. The proposed system consists of an extended Camshift algorithm for the detection of the remote expert surgeon’s hand during surgery and enhancement of an image synthesization algorithm to remove the occlusion by including two Red-Green-Blue-Depth (RGBD) cameras. The experimental results from 10 breast samples reduce the average accuracy error of the video frame image overlay from 1.44 mm to 1.28 mm with average processing time enhanced to 76 sec compared to the state-of-the-art method. The proposed system provides a satisfactory outcome in terms of accuracy and operating time, which allows surgeons in local and remote sites to collaborate more effectively.

[1]  Brent A. Ponce,et al.  The Emergence of Augmented Reality in Orthopaedic Surgery and Education , 2015 .

[2]  Voicu Popescu,et al.  Avoiding Focus Shifts in Surgical Telementoring Using an Augmented Reality Transparent Display , 2016, MMVR.

[3]  Nassir Navab,et al.  Multi-layer Visualization for Medical Mixed Reality , 2017, ArXiv.

[4]  Yan Gao,et al.  Video composition by optimized 3D mean‐value coordinates , 2012, Comput. Animat. Virtual Worlds.

[5]  Yongtian Wang,et al.  Mixed and Augmented Reality Innovations in Beijing Institute of Technology , 2016 .

[6]  Jaromir Przybylo.  Hand tracking algorithm for augmented reality systems , 2010 .

[7]  Alfredo Quinones-Hinojosa,et al.  Real-time Video-Streaming to Surgical Loupe Mounted Head-Up Display for Navigated Meningioma Resection. , 2017, Turkish neurosurgery.

[8]  Aurélio J. C. Campilho,et al.  Real-time implementation of an optical flow algorithm , 2002, Object recognition supported by user interaction for service robots.

[9]  VISION-BASED THREE DIMENSIONAL HAND INTERACTION IN MARKERLESS AUGMENTED REALITY ENVIRONMENT NG KAH PIN UNIVERSITI SAINS MALAYSIA 2015 VISION-BASED THREE DIMENSIONAL HAND INTERACTION IN MARKERLESS AUGMENTED REALITY ENVIRONMENT by NG KAH PIN Thesis submitted in fulfillment of the requirements for the , 2015 .

[10]  Yanbo Guo,et al.  Commercial Videoconferencing for Use in Telementoring Laparoscopic Surgery , 2014, MMVR.

[11]  D. D. Bruellmann,et al.  An optimized video system for augmented reality in endodontics: a feasibility study , 2012, Clinical Oral Investigations.

[12]  Chung-Hsien Huang,et al.  Interactive Medical Augmented Reality System for Remote Surgical Assistance , 2015 .

[13]  A. Jemal,et al.  Breast cancer statistics, 2017, racial disparity in mortality by state , 2017, CA: a cancer journal for clinicians.

[14]  P. Milgram,et al.  A Taxonomy of Mixed Reality Visual Displays , 1994 .

[15]  Barton L Guthrie,et al.  Virtual interactive presence for real-time, long-distance surgical collaboration during complex microsurgical procedures. , 2014, Journal of neurosurgery.

[16]  D. Louis Collins,et al.  The state of the art of visualization in mixed reality image guided surgery , 2013, Comput. Medical Imaging Graph..

[17]  B. Guthrie,et al.  Virtual Interactive Presence and Augmented Reality (VIPAR) for Remote Surgical Assistance , 2011, Neurosurgery.

[18]  Rong Wang,et al.  Autostereoscopic augmented reality visualization for depth perception in endoscopic surgery , 2017, Displays.

[19]  Gary Bradski,et al.  Computer Vision Face Tracking For Use in a Perceptual User Interface , 1998 .

[20]  Akihiro Sugimoto,et al.  Synthesis of Environment Maps for Mixed Reality , 2017, 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).