Measuring the Latency of an Augmented Reality System for Robot-assisted Minimally Invasive Surgery

Minimal latency is important for augmented reality systems and teleoperation interfaces as even small increases in latency can affect user performance. Previously, we have developed an augmented reality system that can overlay stereoscopic video streams with computer graphics in order to improve visual communication in training for robot-assisted minimally invasive surgery with da Vinci surgical systems. To make sure that our augmented reality system provides the best possible user experience, we investigated the video latency of the da Vinci surgical system and how the components of our system affect the overall latency. To measure the photon-to-photon latency, we used a microcontroller to determine the time between the activation of a lightemitting diode in front of the endoscopic camera and the corresponding increase in intensity of the surgeon’s display as measured by a phototransistor. The latency of the da Vinci S surgical system was on average 62 ms. None of the components of our overlay system (separately or combined) significantly affected the latency. However, the latency of the assistant’s monitor increased by 14 ms. Passing the video streams through CPU or GPU memory increased the latency to 147 ms and 256 ms, respectively.

[1]  Fredrik Bruzelius,et al.  A Technical Platform using Augmented Reality for Active Safety Testing , 2015 .

[2]  Martin Møller Jensen,et al.  Stereoscopic augmented reality system for supervised training on minimal invasive surgery robots , 2014, VRIC.

[3]  Bernard D. Adelstein,et al.  Factors influencing operator interaction with virtual objects viewed via head-mounted see-through displays: viewing conditions and rendering latency , 1997, Proceedings of IEEE 1997 Annual International Symposium on Virtual Reality.

[4]  Ronald Azuma,et al.  Recent Advances in Augmented Reality , 2001, IEEE Computer Graphics and Applications.

[5]  Eckehard G. Steinbach,et al.  A System for Precise End-to-End Delay Measurements in Video Communication , 2015, ArXiv.

[6]  M. Kraus,et al.  Real-Time Augmented Reality for Robotic-Assisted Surgery , 2015 .

[7]  Inderbir S. Gill,et al.  Beyond 2D telestration: an evaluation of novel proctoring tools for robot-assisted minimally invasive surgery , 2016, Journal of Robotic Surgery.

[8]  Mark A. Livingston,et al.  Managing latency in complex augmented reality systems , 1997, SI3D.

[9]  Russell H. Taylor,et al.  Augmented reality during robot-assisted laparoscopic partial nephrectomy: toward real-time 3D-CT to stereoscopic video registration. , 2009, Urology.

[10]  Michael Figl,et al.  Image guidance for robotic minimally invasive coronary artery bypass , 2010, Comput. Medical Imaging Graph..

[11]  Ravin Balakrishnan,et al.  Reaching for objects in VR displays: lag and frame rate , 1994, TCHI.

[12]  Eckehard G. Steinbach,et al.  A system for high precision glass-to-glass delay measurements in video communication , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[13]  Martin Kraus,et al.  Pointing with a One-Eyed Cursor for Supervised Training in Minimally Invasive Robotic Surgery , 2015, CARE@MICCAI.

[14]  Nassir Navab,et al.  Measurement of absolute latency for video see through augmented reality , 2007, 2007 6th IEEE and ACM International Symposium on Mixed and Augmented Reality.

[15]  Peter Yellowlees,et al.  3-D telestration: a teaching tool for robotic surgery. , 2008, Journal of laparoendoscopic & advanced surgical techniques. Part A.

[16]  Naoki Suzuki,et al.  A robotic surgery system (da Vinci) with image guided function--system architecture and cholecystectomy application. , 2003, Studies in health technology and informatics.