Time-of-flight-assisted Kinect camera-based people detection for intuitive human robot cooperation in the surgical operating room

BackgroundScene supervision is a major tool to make medical robots safer and more intuitive. The paper shows an approach to efficiently use 3D cameras within the surgical operating room to enable for safe human robot interaction and action perception. Additionally the presented approach aims to make 3D camera-based scene supervision more reliable and accurate.MethodsA camera system composed of multiple Kinect and time-of-flight cameras has been designed, implemented and calibrated. Calibration and object detection as well as people tracking methods have been designed and evaluated.ResultsThe camera system shows a good registration accuracy of 0.05 m. The tracking of humans is reliable and accurate and has been evaluated in an experimental setup using operating clothing. The robot detection shows an error of around 0.04 m.ConclusionsThe robustness and accuracy of the approach allow for an integration into modern operating room. The data output can be used directly for situation and workflow detection as well as collision avoidance.

[1]  Giancarlo Ferrigno,et al.  Event-based device-behavior switching in surgical human-robot interaction , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[2]  Alin Albu-Schäffer,et al.  The DLR MiroSurge - A robotic system for surgery , 2009, 2009 IEEE International Conference on Robotics and Automation.

[3]  Heinz Wörn,et al.  A Novel 3D Camera Based Supervision System for Safe Human-Robot Interaction in the Operating Room , 2015 .

[4]  John Kenneth Salisbury,et al.  The Intuitive/sup TM/ telesurgery system: overview and application , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[5]  Uwe D. Hanebeck,et al.  Intelligent sensor-scheduling for multi-kinect-tracking , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Heinz Wörn,et al.  Ein KinectTM basiertes Überwachungssystem für Workflowerkennung und Gestensteuerung im Operationssaal , 2012, CURAC.

[7]  Heinz Wörn,et al.  OP:Sense — An integrated rapid development environment in the context of robot assisted surgery and operation room sensing , 2011, 2011 IEEE International Conference on Robotics and Biomimetics.

[8]  James H. Anderson,et al.  A robotic system for percutaneous renal access. , 1997, The Journal of urology.

[9]  Branko Karan.  Calibration of Kinect-type RGB-D sensors for robotic applications , 2015 .

[10]  Yongquan Chen,et al.  A Multi-User Interaction System Based on Kinect and Wii Remote , 2012, 2012 IEEE International Conference on Multimedia and Expo Workshops.

[11]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[12]  Hrvoje Benko,et al.  Combining multiple depth cameras and projectors for interactions on, above and between surfaces , 2010, UIST.

[13]  Mario Cifrek,et al.  A brief introduction to OpenCV , 2012, 2012 Proceedings of the 35th International Convention MIPRO.

[14]  Daniel Cremers,et al.  Real-time human motion tracking using multiple depth cameras , 2012, 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[15]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[16]  Alin Albu-Schäffer,et al.  The KUKA-DLR Lightweight Robot arm - a new reference platform for robotics research and manufacturing , 2010, ISR/ROBOTIK.

[17]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[18]  Yasushi Makihara,et al.  Dynamic scene reconstruction using asynchronous multiple Kinects , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[19]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[20]  Heinz Wörn,et al.  The OP : Sense Surgical Robotics Platform : First Feasibility Studies and Current Research , 2013 .

[21]  Blake Hannaford,et al.  Raven-II: An Open Platform for Surgical Robotics Research , 2013, IEEE Transactions on Biomedical Engineering.

[22]  Yao Zhang,et al.  Parallel Computing Experiences with CUDA , 2008, IEEE Micro.

[23]  X. Pennec.  Computing the Mean of Geometric Features Application to the Mean Rotation , 1998 .

[24]  Elena De Momi,et al.  Multi kinect people detection for intuitive and safe human robot cooperation in the operating room , 2013, 2013 16th International Conference on Advanced Robotics (ICAR).

[25]  Andrew W. Fitzgibbon,et al.  Real-time human pose recognition in parts from single depth images , 2011, CVPR 2011.

[26]  Paul J. Besl,et al.  A Method for Registration of 3-D Shapes , 1992, IEEE Trans. Pattern Anal. Mach. Intell..