Real-Time Range Imaging in Health Care: A Survey

The recent availability of dynamic, dense, and low-cost range imaging has gained widespread interest in health care. It opens up new opportunities and has an increasing impact on both research and commercial activities. This chapter presents a state-of-the-art survey on the integration of modern range imaging sensors into medical applications. The scope is to identify promising applications and methods, and to provide an overview of recent developments in this rapidly evolving domain. The survey covers a broad range of topics, including guidance in computer-assisted interventions, operation room monitoring and workflow analysis, touch-less interaction and on-patient visualization, as well as prevention and support in elderly care and rehabilitation. We put emphasis on dynamic and interactive tasks where real-time and dense 3-D imaging forms the key aspect. While considering different range imaging modalities that fulfill these requirements, we particularly investigate the impact of Time-of-Flight imaging in this domain. Eventually, we discuss practical demands and limitations, and open research issues and challenges that are of fundamental importance for the progression of the field.

[1]  Gérard G. Medioni,et al.  Object modelling by registration of multiple range images , 1992, Image Vis. Comput..

[2]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[3]  Thomas Martin Deserno,et al.  Bildverarbeitung für die Medizin: Grundlagen, Modelle, Methoden, Anwendungen , 1997, Bildverarbeitung für die Medizin.

[4]  A. Roggan,et al.  Optical Properties of Circulating Human Blood in the Wavelength Range 400-2500 nm. , 1999, Journal of biomedical optics.

[5]  T. Peters,et al.  An algorithmic overview of surface registration techniques for medical imaging , 2000, Medical Image Anal..

[6]  Benoit M Dawant,et al.  Incorporation of a laser range scanner into image-guided liver surgery: surface acquisition, registration, and tracking. , 2003, Medical physics.

[7]  C Baur,et al.  A non-contact mouse for surgeon-computer interaction. , 2004, Technology and health care : official journal of the European Society for Engineering and Medicine.

[8]  A robust visual tracking system for patient motion detection in SPECT: hardware solutions , 2005, IEEE Symposium Conference Record Nuclear Science 2004..

[9]  Marco Nolden,et al.  The Medical Imaging Interaction Toolkit , 2005, Medical Image Anal..

[10]  Robert L. Galloway,et al.  Compensating for intraoperative soft-tissue deformations using incomplete surface data and finite elements , 2005, IEEE Transactions on Medical Imaging.

[11]  Luis Ibáñez,et al.  The ITK Software Guide , 2005 .

[12]  Guido Gerig,et al.  Medical Image Computing and Computer-Assisted Intervention - MICCAI 2005, 8th International Conference, Palm Springs, CA, USA, October 26-29, 2005, Proceedings, Part II , 2005, MICCAI.

[13]  Guang-Zhong Yang,et al.  Soft-Tissue Motion Tracking and Structure Estimation for Robotic Assisted MIS Procedures , 2005, MICCAI.

[14]  George T. Y. Chen,et al.  A phantom evaluation of a stereo-vision surface imaging system for radiotherapy patient setup. , 2005, Medical physics.

[15]  F. Yin,et al.  The correlation evaluation of a tumor tracking system using multiple external markers. , 2006, Medical physics.

[16]  Robert Lange,et al.  3D time-of-flight distance measurement with custom solid-state image sensors in CMOS/CCD-technology , 2006 .

[17]  Steve B. Jiang,et al.  The management of respiratory motion in radiation oncology report of AAPM Task Group 76. , 2006, Medical physics.

[18]  David Fofi,et al.  A review of recent range image registration methods with accuracy evaluation , 2007, Image Vis. Comput..

[19]  Logan W. Clements,et al.  Concepts and Preliminary Data Toward the Realization of Image-guided Liver Surgery , 2007, Journal of Gastrointestinal Surgery.

[20]  W. Schlegel,et al.  Accuracy of a commercial optical 3D surface imaging system for realignment of patients for radiotherapy of the thorax , 2007, Physics in medicine and biology.

[21]  Nassir Navab,et al.  Action- and Workflow-Driven Augmented Reality for Computer-Aided Medical Procedures , 2007, IEEE Computer Graphics and Applications.

[22]  Beat P. Müller-Stich,et al.  Soft tissue navigation for laparoscopic partial nephrectomy , 2008, International Journal of Computer Assisted Radiology and Surgery.

[23]  D. Falie,et al.  Respiratory motion visualization and the sleep apnea diagnosis with the time of flight (ToF) camera , 2008 .

[24]  Luc Soler,et al.  A structured light system to guide percutaneous punctures in interventional radiology , 2008, SPIE Photonics Europe.

[25]  Joachim Hornegger,et al.  3-D gesture-based scene navigation in medical imaging applications using Time-of-Flight cameras , 2008, 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops.

[26]  Nassir Navab,et al.  Real-Time 3D Reconstruction for Collision Avoidance in Interventional Environments , 2008, MICCAI.

[27]  Gabor Fichtinger,et al.  Medical Image Computing and Computer-Assisted Intervention - MICCAI 2008, 11th International Conference, New York, NY, USA, September 6-10, 2008, Proceedings, Part I , 2008, International Conference on Medical Image Computing and Computer-Assisted Intervention.

[28]  N. A. Bradley,et al.  Assistive Technology For Visually Impaired And Blind People , 2008 .

[29]  A. Brahme,et al.  4D laser camera for accurate patient positioning, collision avoidance, image fusion and adaptive approaches during diagnostic and therapeutic procedures. , 2008, Medical physics.

[30]  J. Hornegger,et al.  Time-of-flight sensor for respiratory motion gating. , 2008, Medical physics.

[31]  Armin Schneider,et al.  Time-of-Flight 3-D Endoscopy , 2009, MICCAI.

[32]  Joyeeta Mitra Mukherjee,et al.  A flexible multicamera visual-tracking system for detecting and correcting motion-induced artifacts in cardiac SPECT slices. , 2009, Medical physics.

[33]  Joachim Hornegger,et al.  Inverse C-arm Positioning for Interventional Procedures Using Real-Time Body Part Detection , 2009, MICCAI.

[34]  Knut-Andreas Lie,et al.  Scale Space and Variational Methods in Computer Vision, Second International Conference, SSVM 2009, Voss, Norway, June 1-5, 2009. Proceedings , 2009, SSVM.

[35]  Hans-Peter Meinzer,et al.  Statistical shape models for 3D medical image segmentation: A review , 2009, Medical Image Anal..

[36]  Guang-Zhong Yang,et al.  Medical Image Computing and Computer-Assisted Intervention - MICCAI 2009, 12th International Conference, London, UK, September 20-24, 2009, Proceedings, Part II , 2009, MICCAI.

[37]  Joachim Hornegger,et al.  Time-of-Flight sensor for patient positioning , 2009, Medical Imaging.

[38]  Nassir Navab,et al.  Workflow monitoring based on 3D motion features , 2009, 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops.

[39]  Kevin Wells,et al.  An observation model for motion correction in nuclear medicine , 2010, Medical Imaging.

[40]  T. Gevaert,et al.  Gating and tracking, 4D in thoracic tumours. , 2010, Cancer radiotherapie : journal de la Societe francaise de radiotherapie oncologique.

[41]  Guang-Zhong Yang,et al.  Three-Dimensional Tissue Deformation Recovery and Tracking , 2010, IEEE Signal Processing Magazine.

[42]  Wolfgang Kemmler,et al.  Measurement of angles in time-of-flight data for the automatic supervision of training exercises , 2010, 2010 4th International Conference on Pervasive Computing Technologies for Healthcare.

[43]  R. Gassert,et al.  Augmented white cane with multimodal haptic feedback , 2010, 2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics.

[44]  Nassir Navab,et al.  Estimating Radiation Exposure in Interventional Environments , 2010, MICCAI.

[45]  Sebastian Thrun,et al.  Real time motion capture using a single time-of-flight camera , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[46]  Lena Maier-Hein,et al.  Accounting for Anisotropic Noise in Fine Registration of Time-of-Flight Range Data with High-Resolution Surface Data , 2010, MICCAI.

[47]  Stefan Fuchs,et al.  Multipath Interference Compensation in Time-of-Flight Camera Images , 2010, 2010 20th International Conference on Pattern Recognition.

[48]  Rasmus Larsen,et al.  Structured light 3D tracking system for measuring motions in PET brain imaging , 2010, Medical Imaging.

[49]  R. Higuchi,et al.  Image overlay navigation by markerless surface registration in gastrointestinal, hepatobiliary and pancreatic surgery , 2010, Journal of hepato-biliary-pancreatic sciences.

[50]  Nassir Navab,et al.  Medical Image Computing and Computer-Assisted Intervention - MICCAI 2010, 13th International Conference, Beijing, China, September 20-24, 2010, Proceedings, Part III , 2010, MICCAI.

[51]  Yao-Jen Chang,et al.  A Kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities. , 2011, Research in developmental disabilities.

[52]  Benjamin Berkels,et al.  Joint ToF Image Denoising and Registration with a CT Surface in Radiation Therapy , 2011, SSVM.

[53]  Heinz Wörn,et al.  A Semi-Autonomous Robotic Teleoperation Surgery Setup with Multi 3D Camera Supervision , 2011 .

[54]  Tinsu Pan,et al.  Author manuscript, published in "Journal of Medical Physics 2011;38(6):3157-3164" DOI: 10.1118/1.3589131 Technical Note: Correlation of respiratory motion between external patient surface and internal anatomical landmarks , 2011 .

[55]  Miriam Vollenbroek-Hutten,et al.  Chronic pain rehabilitation with a serious game using multimodal input , 2011, 2011 International Conference on Virtual Rehabilitation.

[56]  Mario Ciampi,et al.  Controller-free exploration of medical image data: Experiencing the Kinect , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[57]  Marjorie Skubic,et al.  Evaluation of an inexpensive depth camera for in-home gait assessment , 2011, J. Ambient Intell. Smart Environ..

[58]  Horst-Michael Groß,et al.  Progress in developing a socially assistive mobile home robot companion for the elderly with mild cognitive impairment , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[59]  Guang-Zhong Yang,et al.  An endoscopic structured lighting probe using spectral encoding , 2011, European Conference on Biomedical Optics.

[60]  Russell H. Taylor,et al.  Information Processing in Computer-Assisted Interventions - Second International Conference, IPCAI 2011, Berlin, Germany, June 22, 2011. Proceedings , 2011, IPCAI.

[61]  Jun-Da Huang.  Kinerehab: a kinect-based system for physical rehabilitation: a pilot study for young adults with motor disabilities , 2011, ASSETS '11.

[62]  Marco Nolden,et al.  MITK-ToF—Range data within MITK , 2011, International Journal of Computer Assisted Radiology and Surgery.

[63]  Ghassan Hamarneh,et al.  A Survey on Shape Correspondence , 2011, Comput. Graph. Forum.

[64]  G. Strauss,et al.  Intraoperative Bedienung einer elektronischen Patientenakte durch den Operateur , 2011, HNO.

[65]  Lena Maier-Hein,et al.  3D surface reconstruction for laparoscopic computer-assisted interventions: comparison of state-of-the-art methods , 2011, Medical Imaging.

[66]  Lena Maier-Hein,et al.  Time-of-flight camera technique for augmented reality in computer-assisted interventions , 2011, Medical Imaging.

[67]  M. Riboldi,et al.  A multiple points method for 4D CT image sorting. , 2011, Medical physics.

[68]  Gregory Hager,et al.  Vision-based navigation in image-guided interventions. , 2011, Annual review of biomedical engineering.

[69]  Lena Maier-Hein,et al.  Robust multi-modal surface matching for intra-operative registration , 2011, 2011 24th International Symposium on Computer-Based Medical Systems (CBMS).

[70]  Sven Haase,et al.  Multi-modal surface registration for markerless initial patient setup in radiation therapy using microsoft's Kinect sensor , 2011, 2011 IEEE International Conference on Computer Vision Workshops (ICCV Workshops).

[71]  Albert A. Rizzo,et al.  Development and evaluation of low cost game-based balance rehabilitation tool using the microsoft kinect sensor , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[72]  Nassir Navab,et al.  Estimating human 3D pose from Time-of-Flight images based on geodesic distances and optical flow , 2011, Face and Gesture 2011.

[73]  Marjorie Skubic,et al.  Passive in-home measurement of stride-to-stride gait variability comparing vision and Kinect sensing , 2011, 2011 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[74]  Sven Haase,et al.  RITK: The Range Imaging Toolkit - A Framework for 3-D Range Image Stream Processing , 2011, VMV.

[75]  Lena Maier-Hein,et al.  Towards Mobile Augmented Reality for On-Patient Visualization of Medical Images , 2011, Bildverarbeitung für die Medizin.

[76]  Michael J. Cree,et al.  Separating true range measurements from multi-path and scattering interference in commercial range cameras , 2011, Electronic Imaging.

[77]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[78]  John Philbeck,et al.  Evaluation of gesture based interfaces for medical volume visualization tasks , 2011, VRCAI '11.

[79]  Luis Alvarez,et al.  Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications , 2012, Lecture Notes in Computer Science.

[80]  Benjamin Berkels,et al.  Marker-Less Reconstruction of Dense 4-D Surface Motion Fields Using Active Laser Triangulation for Respiratory Motion Management , 2012, MICCAI.

[81]  Günther Greiner,et al.  Markerless estimation of patient orientation, posture and pose using range and pressure imaging , 2012, International Journal of Computer Assisted Radiology and Surgery.

[82]  Jakob Wasza,et al.  Real-Time Motion Compensated Patient Positioning and Non-rigid Deformation Estimation Using 4-D Shape Priors , 2012, MICCAI.

[83]  Vassilios Morellas,et al.  A multi-sensor visual tracking system for behavior monitoring of at-risk children , 2012, 2012 IEEE International Conference on Robotics and Automation.

[84]  A. Seitel.  Markerless navigation for percutaneous needle insertions , 2012 .

[85]  Matthew O'Toole,et al.  Decomposing global light transport using time of flight imaging , 2012, CVPR.

[86]  Guilherme C. S. Ruppert,et al.  Touchless gesture user interface for interactive image visualization in urological surgery , 2012, World Journal of Urology.

[87]  Antonio Coronato,et al.  Towards abnormal behavior detection of cognitive impaired people , 2012, 2012 IEEE International Conference on Pervasive Computing and Communications Workshops.

[88]  Ming-Sui Lee,et al.  Breath and Position Monitoring during Sleeping with a Depth Camera , 2012, HEALTHINF.

[89]  Albert A. Rizzo,et al.  Towards pervasive physical rehabilitation using Microsoft Kinect , 2012, 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops.

[90]  Jon Howard,et al.  Realtime markerless rigid body head motion tracking using the Microsoft Kinect , 2012, 2012 IEEE Nuclear Science Symposium and Medical Imaging Conference Record (NSS/MIC).

[91]  Michel Denis,et al.  NAVIG: augmented reality guidance system for the visually impaired , 2012, Virtual Reality.

[92]  Russell H. Taylor,et al.  Towards Automated Activity Recognition in an Intensive Care Unit , 2012 .

[93]  Lena Maier-Hein,et al.  Convergent Iterative Closest-Point Algorithm to Accomodate Anisotropic and Inhomogenous Localization Error , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[94]  David Sarrut,et al.  Multi-dimensional respiratory motion tracking from markerless optical surface imaging based on deformable mesh registration , 2012, Physics in medicine and biology.

[95]  Sven Haase,et al.  Sparse Principal Axes Statistical Surface Deformation Models for Respiration Analysis and Classification , 2012, Bildverarbeitung für die Medizin.

[96]  Keno März,et al.  Registration of partially overlapping surfaces for range image based augmented reality on mobile devices , 2012, Medical Imaging.

[97]  Hervé Delingette,et al.  Medical Image Computing and Computer-Assisted Intervention – MICCAI 2012 , 2012, Lecture Notes in Computer Science.

[98]  Fatemeh Tahavori,et al.  A quantitative assessment of using the Kinect for Xbox 360 for respiratory surface motion tracking , 2012, Medical Imaging.

[99]  Alex Mihailidis,et al.  3D Human Motion Analysis to Detect Abnormal Events on Stairs , 2012, 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization & Transmission.

[100]  Jon Howard,et al.  Accurate markerless respiratory tracking for gated whole body PET using the Microsoft Kinect , 2012, 2012 IEEE Nuclear Science Symposium and Medical Imaging Conference Record (NSS/MIC).

[101]  Juan Pablo Wachs,et al.  Intention, Context and Gesture Recognition for Sterile MRI Navigation in the Operating Room , 2012, CIARP.

[102]  Michael J Thali,et al.  You Can’t Touch This , 2012, Surgical innovation.

[103]  Jakob Wasza,et al.  Photometric Estimation of 3D Surface Motion Fields for Respiration Management , 2012, Bildverarbeitung für die Medizin.

[104]  Nassir Navab,et al.  mirracle: An augmented reality magic mirror system for anatomy education , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[105]  Nassir Navab,et al.  Simultaneous categorical and spatio-temporal 3D gestures using Kinect , 2012, 2012 IEEE Symposium on 3D User Interfaces (3DUI).

[106]  Veronica Teichrieb,et al.  Poster: Improving motor rehabilitation process through a natural interaction based system using Kinect sensor , 2012, 2012 IEEE Symposium on 3D User Interfaces (3DUI).

[107]  Ran Gilad-Bachrach,et al.  Full body gait analysis with Kinect , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[108]  Yusuf Pisan,et al.  Exergames for the elderly: Towards an embedded Kinect-based clinical test of falls risk , 2012, HIC.

[109]  Vassilios Morellas,et al.  A nonintrusive system for behavioral analysis of children using multiple RGB+depth sensors , 2012, 2012 IEEE Workshop on the Applications of Computer Vision (WACV).

[110]  Adrien Bartoli,et al.  Towards Live Monocular 3D Laparoscopy Using Shading and Specularity Information , 2012, IPCAI.

[111]  Christoph Schmalz,et al.  An endoscopic 3D scanner based on structured light , 2012, Medical Image Anal..

[112]  Gregory D. Hager,et al.  The Kinect as an interventional tracking system , 2012, Medical Imaging.

[113]  D. Sharma,et al.  Senior health monitoring using Kinect , 2012, 2012 Fourth International Conference on Communications and Electronics (ICCE).

[114]  Henry Fuchs,et al.  Reducing interference between multiple structured light depth sensors using motion , 2012, 2012 IEEE Virtual Reality Workshops (VRW).

[115]  N. Kubota,et al.  Robot Partner System for elderly people care by using sensor network , 2012, 2012 4th IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[116]  Elli Angelopoulou,et al.  Fast time-of-flight camera based surface registration for radiotherapy patient positioning. , 2011, Medical physics.

[117]  Stuart T. Smith,et al.  The use of exercise-based videogames (exergames) for training and rehabilitation of physical function in older adults: current practice and guidelines for future research , 2012 .

[118]  Adrien Bartoli,et al.  Template-Based Conformal Shape-from-Motion-and-Shading for Laparoscopy , 2012, IPCAI.

[119]  Junyi Xia,et al.  A real-time respiratory motion monitoring system using KINECT: proof of concept. , 2012, Medical physics.

[120]  Sebastian Bauer,et al.  Medical Applications enabled by a motion-robust optical 3D sensor , 2012 .

[121]  Ross B. Girshick,et al.  Efficient Human Pose Estimation from Single Depth Images , 2013, IEEE Trans. Pattern Anal. Mach. Intell..

[122]  David J. Hawkes,et al.  Respiratory motion models: a review. , 2013 .

[123]  Sven Haase,et al.  Laparoscopic instrument localization using a 3-D Time-of-Flight/RGB endoscope , 2013, 2013 IEEE Workshop on Applications of Computer Vision (WACV).

[124]  Nikos Paragios,et al.  Deformable Medical Image Registration: A Survey , 2013, IEEE Transactions on Medical Imaging.

[125]  Lena Maier-Hein,et al.  Optical techniques for 3D surface reconstruction in computer-assisted laparoscopic surgery , 2013, Medical Image Anal..

[126]  Stephan Klöck,et al.  TOPOS: a new topometric patient positioning and tracking system for radiation therapy based on structured white light. , 2013, Medical physics.

[127]  Stefanie Speidel,et al.  Ontology-based prediction of surgical events in laparoscopic surgery , 2013, Medical Imaging.

[128]  A Y C Nee,et al.  Assistive obstacle detection and navigation devices for vision-impaired users , 2013, Disability and rehabilitation. Assistive technology.

[129]  V. Bettinardi,et al.  Motion-Tracking Hardware and Advanced Applications in PET and PET/CT. , 2013, PET clinics.

[130]  Christian Nøhr,et al.  Comparing Approaches to Measuring the Adoption and Usability of Electronic Health Records: Lessons Learned from Canada, Denmark and Finland , 2013, MedInfo.