Immersive Learning Experiences for Surgical Procedures

This paper introduces a computer-based system that is designed to record a surgical procedure with multiple depth cameras and reconstruct in three dimensions the dynamic geometry of the actions and events that occur during the procedure. The resulting 3D-plus-time data takes the form of dynamic, textured geometry and can be immersively examined at a later time; equipped with a Virtual Reality headset such as Oculus Rift DK2, a user can walk around the reconstruction of the procedure room while controlling playback of the recorded surgical procedure with simple VCR-like controls (play, pause, rewind, fast forward). The reconstruction can be annotated in space and time to provide more information of the scene to users. We expect such a system to be useful in applications such as training of medical students and nurses.

[1]  Joseph L. Mundy,et al.  Image-Based 4-d Reconstruction Using 3-d Change Detection , 2014, ECCV.

[2]  Jan-Michael Frahm,et al.  Exploring High-Level Plane Primitives for Indoor 3D Reconstruction with a Hand-held RGB-D Camera , 2012, ACCV Workshops.

[3]  Andrew W. Fitzgibbon,et al.  3D scanning deformable objects with a single RGBD sensor , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Marc Pollefeys,et al.  Image based detection of geometric changes in urban environments , 2011, 2011 International Conference on Computer Vision.

[5]  Bennett A Landman,et al.  Immersive virtual reality for visualization of abdominal CT , 2013, Medical Imaging.

[6]  William E. Lorensen,et al.  Marching cubes: A high resolution 3D surface construction algorithm , 1987, SIGGRAPH.

[7]  Marc Levoy,et al.  A volumetric method for building complex models from range images , 1996, SIGGRAPH.

[8]  Henry Fuchs,et al.  Immersive 3D Telepresence , 2014, Computer.

[9]  Dieter Fox,et al.  DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Henry Fuchs,et al.  Temporally enhanced 3D capture of room-sized dynamic scenes with commodity depth cameras , 2014, 2014 IEEE Virtual Reality (VR).

[12]  Patrick Pérez,et al.  Multi-view Object Segmentation in Space and Time , 2013, 2013 IEEE International Conference on Computer Vision.

[13]  Andrew W. Fitzgibbon,et al.  Real-time non-rigid reconstruction using an RGB-D camera , 2014, ACM Trans. Graph..

[14]  Alberto Del Bimbo,et al.  A Natural and Immersive Virtual Interface for the Surgical Safety Checklist Training , 2014, SeriousGames '14.

[15]  Ruzena Bajcsy,et al.  Teleimmersive Environment for Remote Medical Collaboration , 2009, MMVR.

[16]  Uwe Kloos,et al.  Enhancing Medical Communication Training Using Motion Capture, Perspective Taking and Virtual Reality , 2012, MMVR.

[17]  Andrew W. Fitzgibbon,et al.  KinectFusion: Real-time dense surface mapping and tracking , 2011, 2011 10th IEEE International Symposium on Mixed and Augmented Reality.

[18]  Michael J Thali,et al.  The forensic holodeck: an immersive display for forensic crime scene reconstructions , 2014, Forensic Science, Medicine, and Pathology.

[19]  Yuan Liu,et al.  Virtual neurosurgical education for image-guided deep brain stimulation neurosurgery , 2014, 2014 International Conference on Audio, Language and Image Processing.

[20]  Anselmo Lastra,et al.  Immersive electronic books for surgical training , 2005, IEEE MultiMedia.

[21]  Wm. LeRoy Heinrichs,et al.  CliniSpaceTM: A Multiperson 3D Online Immersive Training Environment Accessible through a Browser , 2011, MMVR.