Electronic image stabilization using optical flow with inertial fusion

When a camera is affixed on a dynamic mobile robot, image stabilization is the first step towards more complex analysis on the video feed. This paper presents a novel electronic image stabilization (EIS) algorithm for highly dynamic mobile robotic platforms. The algorithm combines optical flow motion parameter estimation with angular rate data provided by a strapdown inertial measurement unit (IMU). A discrete Kalman filter in feedforward configuration is used for optimal fusion of the two data sources. Performance evaluations are conducted using a simulated video truth model (capturing the effects of image translation, rotation, blurring, and moving objects), and live test data. Live data was collected from a camera and IMU affixed to the DAGSI Whegs mobile robotic platform as it navigated through a hallway. Template matching, feature detection, optical flow, and inertial measurement techniques are compared and analyzed to determine the most suitable algorithm for this specific type of image stabilization. Pyramidal Lucas-Kanade optical flow using Shi-Tomasi good features in combination with inertial measurement is the EIS algorithm found to be superior. In the presence of moving objects, fusion of inertial measurement reduces optical flow root-mean-squared (RMS) error in motion parameter estimates by 40%.

[1]  Mohammad Rahmati,et al.  An electronic digital image stabilizer based on stationary wavelet transform (SWT) , 2003, Proceedings 2003 International Conference on Image Processing (Cat. No.03CH37429).

[2]  Michael Veth,et al.  Fusion of Imaging and Inertial Sensors for Navigation , 2006 .

[3]  Ryo Kurazume,et al.  Development of image stabilization system for remote operation of walking robots , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[4]  Richard A. Brown,et al.  Introduction to random signals and applied kalman filtering (3rd ed , 2012 .

[5]  Mohammed A. Alharbi.  Fast Video Stabilization Algorithms , 2012 .

[6]  Matthijs C. Dorst.  Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[7]  Michael John Smith,et al.  Electronic Image Stabilization for Mobile Robotic Vision Systems , 2012 .

[8]  A. Amanatiadis,et al.  A Rotational and Translational Image Stabilization System for Remotely Operated Robots , 2007, 2007 IEEE International Workshop on Imaging Systems and Techniques.

[9]  Nasser Kehtarnavaz,et al.  Proceedings of SPIE - The International Society for Optical Engineering , 1991 .

[10]  Gilbert L. Peterson,et al.  The latest generation Whegs™ robot features a passive-compliant body joint , 2008, 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[11]  Gary R. Bradski,et al.  Learning OpenCV - computer vision with the OpenCV library: software that sees , 2008 .

[12]  Michael A. Goodrich,et al.  Application and evaluation of spatiotemporal enhancement of live aerial video using temporally local mosaics , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[13]  Yong-xiang Zhang.  A Method of Resolving Gyro Zero Drift in Electronic Stabilization System , 2009, 2009 International Conference on Computer and Automation Engineering.

[14]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[15]  Junyao Gao,et al.  A real-time scheme of video stabilization for mine tunnel inspectional robot , 2007, 2007 IEEE International Conference on Robotics and Biomimetics (ROBIO).

[16]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[17]  Fabio Gagliardi Cozman,et al.  Fast software image stabilization with color registration , 1998, Proceedings. 1998 IEEE/RSJ International Conference on Intelligent Robots and Systems. Innovations in Theory, Practice and Applications (Cat. No.98CH36190).

[18]  Manabu Hashimoto,et al.  Hierarchical distributed template matching , 1997, Electronic Imaging.

[19]  Rama Chellappa,et al.  Fast electronic digital image stabilization , 1996, Proceedings of 13th International Conference on Pattern Recognition.

[20]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[21]  Rama Chellappa,et al.  Stabilization and Mosaicing of Airborne Videos , 2006, 2006 International Conference on Image Processing.