Deeply Learned 2D Tool Pose Estimation for Robot-to-Camera Registration

Robot-assisted eye surgery is the central topic of the EU funded project EurEyeCase. Major objectives of the project comprise the development of methodologies to perform two surgical procedures that cannot be easily carried out by human surgeons, namely retinal vein cannulation and retinal membrane peeling. In the proposed assistive system, visual guidance is provided from a camera mounted on the microscope. In order to guide the robot using visual cues, it is necessary to register the camera coordinates to the robot coordinates. To this end, we propose a framework that estimates the position and the pose of the tool to register the two different coordinate systems. Using recent advances in convolutional neural networks (CNNs), we present a comparative study among different intuitive architectural designs, and suggest a methodology to register the coordinates by detecting pre-defined keypoints. Results suggest that tool pose estimation can be highly accurate, running in real-time on a GPU.