A tutorial on visual servo control

This article provides a tutorial introduction to visual servo control of robotic manipulators. Since the topic spans many disciplines our goal is limited to providing a basic conceptual framework. We begin by reviewing the prerequisite topics from robotics and computer vision, including a brief review of coordinate transformations, velocity representation, and a description of the geometric aspects of the image formation process. We then present a taxonomy of visual servo control systems. The two major classes of systems, position-based and image-based systems, are then discussed in detail. Since any visual servo system must be capable of tracking image features in a sequence of images, we also include an overview of feature-based and correlation-based methods for tracking. We conclude the tutorial with a number of observations on the current directions of the research field of visual servo control.

[1]  D. E. Whitney,et al.  The mathematics of coordinated control of prosthetic arms and manipulators. , 1972 .

[2]  Yoshiaki Shirai,et al.  Guiding a robot by visual feedback in assembling tasks , 1973, Pattern Recognit..

[3]  I. E. Sutherland.  Three-dimensional data input by tablet , 1974, COMG.

[4]  A.H. Haddad,et al.  Applied optimal estimation , 1976, Proceedings of the IEEE.

[5]  Robert C. Bolles,et al.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography , 1981, CACM.

[6]  Takeo Kanade,et al.  An Iterative Image Registration Technique with an Application to Stereo Vision , 1981, IJCAI.

[7]  Lee E. Weiss,et al.  Adaptive Visual Servo Control of Robots , 1983 .

[8]  Sundaram Ganapathy,et al.  Decomposition of transformation matrices for robot vision , 1984, Pattern Recognition Letters.

[9]  Gerald J. Agin.  Calibration and use of a light stripe range sensor mounted on the hand of a robot , 1985, Proceedings. 1985 IEEE International Conference on Robotics and Automation.

[10]  Gene F. Franklin,et al.  Feedback Control of Dynamic Systems , 1986 .

[11]  Berthold K. P. Horn.  Robot vision , 1986, MIT electrical engineering and computer science series.

[12]  John F. Canny,et al.  A Computational Approach to Edge Detection , 1986, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[13]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[14]  Yoshiaki Shirai,et al.  Three-Dimensional Computer Vision , 1987, Symbolic Computation.

[15]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using unit quaternions , 1987 .

[16]  Lee E. Weiss,et al.  Dynamic sensor-based control of robots with visual feedback , 1987, IEEE Journal on Robotics and Automation.

[17]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Steven B. Skaar,et al.  Camera-Space Manipulation , 1987 .

[19]  Russell L. Anderson,et al.  A Robot Ping-Pong Player: Experiments in Real-Time Intelligent Control , 1988 .

[20]  Berthold K. P. Horn,et al.  Closed-form solution of absolute orientation using orthonormal matrices , 1988 .

[21]  Azriel Rosenfeld,et al.  Computer Vision , 1988, Adv. Comput..

[22]  Michel Dhome,et al.  Determination of the Attitude of 3D Objects from a Single Perspective View , 1989, IEEE Trans. Pattern Anal. Mach. Intell..

[23]  Roger Y. Tsai,et al.  A new technique for fully autonomous and efficient 3D robotics hand/eye calibration , 1988, IEEE Trans. Robotics Autom..

[24]  Gerd Hirzinger,et al.  On a Unified Concept for a New Generation of Light-Weight Robots , 1989, ISER.

[25]  Radu Horaud,et al.  An analytic solution for the perspective 4-point problem , 1989, Proceedings CVPR '89: IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[26]  R. L. Andersson.  Dynamic sensing in a ping-pong playing robot , 1989, IEEE Trans. Robotics Autom..

[27]  Olivier D. Faugeras,et al.  Determination of Camera Location from 2-D to 3-D Line and Point Correspondences , 1990, IEEE Trans. Pattern Anal. Mach. Intell..

[28]  Suresh Venkatesan,et al.  Realtime tracking in five degrees of freedom using two writs-mounted laser range finders , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[29]  Steven B. Skaar,et al.  Three-Dimensional Camera Space Manipulation , 1990, Int. J. Robotics Res..

[30]  John E. Howland,et al.  Computer graphics , 1990, IEEE Potentials.

[31]  Tsutomu Kimoto,et al.  Manipulator control with image-based visual servo , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[32]  Dimitris P. Tsakiris,et al.  On the visual mathematics of tracking , 1991, Image Vis. Comput..

[33]  C. S. George Lee,et al.  Weighted selection of image features for resolved rate visual feedback control , 1991, IEEE Trans. Robotics Autom..

[34]  Jean-Claude Latombe,et al.  Robot motion planning , 1970, The Kluwer international series in engineering and computer science.

[35]  Myung Jin Chung,et al.  Concepts of Augmented Image Space and Transformed Feature Space for Efficient Visual Servoing of an "Eye-in-Hand Robot" , 1991, Robotica.

[36]  Patrick Rives,et al.  A new approach to visual servoing in robotics , 1992, IEEE Trans. Robotics Autom..

[37]  Robert M. Haralick,et al.  Analysis and solutions of the three point perspective pose estimation problem , 1991, Proceedings. 1991 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[38]  Claude Samson,et al.  Robot Control: The Task Function Approach , 1991 .

[39]  Linda G. Shapiro,et al.  Computer and Robot Vision , 1991 .

[40]  Phillip J. McKerrow,et al.  Introduction to robotics , 1991 .

[41]  David G. Lowe,et al.  Fitting Parameterized Three-Dimensional Models to Images , 1991, IEEE Trans. Pattern Anal. Mach. Intell..

[42]  Bruno Siciliano,et al.  Control of robotic systems through singularities , 1991 .

[43]  Z. Zenn Bien,et al.  Feature-based visual servoing of an eye-in-hand robot with improved tracking performance , 1991, Proceedings. 1991 IEEE International Conference on Robotics and Automation.

[44]  Steven B. Skaar,et al.  Nonholonomic camera-space manipulation , 1992, IEEE Trans. Robotics Autom..

[45]  Larry S. Davis,et al.  Exact and Approximate Solutions of the Perspective-Three-Point Problem , 1992, IEEE Trans. Pattern Anal. Mach. Intell..

[46]  Takeo Kanade,et al.  Visual tracking of a moving target by a camera mounted on a robot: a combination of control and vision , 1993, IEEE Trans. Robotics Autom..

[47]  W. Sardha Wijesoma,et al.  Eye-to-Hand Coordination for Vision-Guided Robot Control Applications , 1993, Int. J. Robotics Res..

[48]  Peter K. Allen,et al.  Automated tracking and grasping of a moving object with a robotic hand-eye system , 1993, IEEE Trans. Robotics Autom..

[49]  Nikolaos Papanikolopoulos,et al.  Adaptive robotic visual tracking: theory and experiments , 1993, IEEE Trans. Autom. Control..

[50]  Peter Corke,et al.  VISUAL CONTROL OF ROBOT MANIPULATORS – A REVIEW , 1993 .

[51]  Seth Hutchinson,et al.  Multi-rate analysis and design of visual feedback digital servo control system , 1993 .

[52]  Pradeep K. Khosla,et al.  Integrating Sensor Placement and Visual Tracking Strategies , 1993, ISER.

[53]  William L. Wilson,et al.  VISUAL SERVO CONTROL OF ROBOTS USING KALMAN FILTER ESTIMATES OF ROBOT POSE RELATIVE TO WORK-PIECES , 1993 .

[54]  Roberto Cipolla,et al.  Uncalibrated stereo hand-eye coordination , 1994, Image Vis. Comput..

[55]  Gregory D. Hager,et al.  Feature-based visual servoing and its application to telerobotics , 1994, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'94).

[56]  Carlo Tomasi,et al.  Good features to track , 1994, 1994 Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.

[57]  Minoru Asada,et al.  Versatile visual servoing without knowledge of true Jacobian , 1994, Proceedings of IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'94).

[58]  Rajeev Sharma,et al.  On the observability of robot motion under active camera control , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[59]  Thomas S. Huang,et al.  Motion and structure from feature correspondences: a review , 1994, Proc. IEEE.

[60]  Ernst Dieter Dickmanns,et al.  Visual grasping with long delay time of a free floating object in orbit , 1994 .

[61]  Peter Corke,et al.  High-Performance Visual Closed-Loop Robot Control , 1994 .

[62]  Allen R. Hanson,et al.  Robust methods for estimating pose and a sensitivity analysis , 1994 .

[63]  Ernst D. Dickmanns,et al.  Visual grasping with long delay time of a free floating object in orbit , 1994, Auton. Robots.

[64]  Seth Hutchinson,et al.  Visual compliance: task-directed visual servo control , 1994, IEEE Trans. Robotics Autom..

[65]  Peter K. Allen,et al.  Active, uncalibrated visual servoing , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[66]  Emanuele Trucco,et al.  Computer and Robot Vision , 1995 .

[67]  Gregory D. Hager,et al.  Robot hand-eye coordination based on stereo vision , 1995 .

[68]  Gregory D. Hager,et al.  Calibration-free visual control using projective invariance , 1995, Proceedings of IEEE International Conference on Computer Vision.

[69]  Armando Fox,et al.  Exploiting visual constraints in the synthesis of uncertainty-tolerant motion plans , 1995, IEEE Trans. Robotics Autom..

[70]  Gregory D. Hager,et al.  A "robust" convergent visual servoing system , 1995, Proceedings 1995 IEEE/RSJ International Conference on Intelligent Robots and Systems. Human Robot Interaction and Cooperative Robots.

[71]  Daniel E. Koditschek,et al.  An active visual estimator for dexterous manipulation , 1996, IEEE Trans. Robotics Autom..

[72]  Peter I. Corke,et al.  Dynamic effects in visual closed-loop systems , 1996, IEEE Trans. Robotics Autom..

[73]  E. Mjolsness,et al.  Online computation of exterior orientation with application to hand-eye calibration , 1996 .

[74]  Mark W. Spong,et al.  Camera modelling for visual servo control applications , 1996 .