Ultra-low-latency endoscopic image stabilisation

During laparoscopic surgery, it is often hard to keep track of the orientation of the endoscope, making the images difficult to interpret. This is particularly true for endoscopes using an angled lens (typically 30 degrees). When used to look behind an object the endoscope is rotated to look left or right causing the image to rotate as well. Currently this problem is circumvented by rotating only the endoscopic lens while keeping the camera steady. This is often cumbersome, requiring both hands of the operator, and restricts camera movement. We propose a system that automatically compensates for the rotation of the camera. Our system is completely independent of the endoscope and works image-based. The rotation estimation and adjustment is done entirely in software using only the video from the endoscope. Our method keeps the latency from camera to screen at its absolute minimum. We use a Kanade–Lucas–Tomasi (KLT) tracker and RANSAC to estimate the image rotation and smooth out and predict the movement using a Kalman filter. Using this approach we are able to accurately stabilise video while still being able to render at minimal latency (¡ 30ms). The amount of drift (movement away from reference rotation) and rotation latency is greatly influenced by the rate at which the angle is measured. Using an off-the-shelf desktop computer our algorithm is able to calculate a new angle 60 times per second. Our embedded implementation is still able to smoothly render video at full framerate using only 8 rotation updates per second. We use a Kalman filter to predict the angle in-between updates and smooth out sudden updates. A video of the result can be found at: http://youtu.be/2tMk0u0yWc0.