Eye-in-Hand Visual Servoing Enhanced With Sparse Strain Measurement for Soft Continuum Robots

In the feature/object tracking of eye-in-hand visual servoing, 2D motion estimation relying only on image plane feedback is easily affected by vision occlusion, blurring, or poor lighting. For the commonly-used template matching method, tracking performance greatly depends on the image quality. Fiber Bragg gratings (FBGs), a type of high-frequency flexible strain sensor, can be used as an assistant device for soft robot control. We propose a method to enhance motion estimation in soft robotic visual servoing by fusing the results from template matching and FBG wavelength shifts to achieve more accurate tracking in applications such as minimally invasive surgery. Path following performance is validated in a simulated laparoscopic scene and LEGO-constructed scene, demonstrating significant improvement to feature tracking and robot motion, even under external forces.

[1]  Peter I. Corke,et al.  A tutorial on visual servo control , 1996, IEEE Trans. Robotics Autom..

[2]  Xiaoping Lou,et al.  FBG based shape sensing of a silicone octopus tentacle model for soft robotics , 2018, Optik.

[3]  Rajnikant V. Patel,et al.  Curvature, Torsion, and Force Sensing in Continuum Robots Using Helically Wrapped FBG Sensors , 2016, IEEE Robotics and Automation Letters.

[4]  Gerhard P. Hancke,et al.  Pose Estimation of a Mobile Robot Based on Fusion of IMU Data and Vision Data Using an Extended Kalman Filter , 2017, Sensors.

[5]  Toshio Fukuda,et al.  Shape Sensing Techniques for Continuum Robots in Minimally Invasive Surgery: A Survey , 2017, IEEE Transactions on Biomedical Engineering.

[6]  Xinwu Liang,et al.  Visual Servoing of Soft Robot Manipulator in Constrained Environments With an Adaptive Controller , 2017, IEEE/ASME Transactions on Mechatronics.

[7]  R. F. Shepherd,et al.  Soft optoelectronic sensory foams with proprioception , 2018, Science Robotics.

[8]  Tao Deng,et al.  Visual servo control of cable-driven soft robotic manipulator , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[9]  Koichi Hashimoto,et al.  Visual Servoing: Real-Time Control of Robot Manipulators Based on Visual Sensory Feedback , 1993 .

[10]  Allison M. Okamura,et al.  Series pneumatic artificial muscles (sPAMs) and application to a soft continuum robot , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[11]  Kit-Hang Lee,et al.  Real-Time Surface Shape Sensing for Soft and Flexible Structures Using Fiber Bragg Gratings , 2019, IEEE Robotics and Automation Letters.

[12]  Kaspar Althoefer,et al.  Nonparametric Online Learning Control for Soft Continuum Robot: An Enabling Technique for Effective Endoscopic Navigation , 2017, Soft robotics.

[13]  Russell H. Taylor,et al.  Shape Tracking of a Dexterous Continuum Manipulator Utilizing Two Large Deflection Shape Sensors , 2015, IEEE Sensors Journal.

[14]  Nima Najmaei,et al.  Visual servoing in medical robotics: a survey. Part I: endoscopic and direct vision imaging – techniques and applications , 2014, The international journal of medical robotics + computer assisted surgery : MRCAS.

[15]  Mehran Armand,et al.  FBG-Based Position Estimation of Highly Deformable Continuum Manipulators: Model-Dependent vs. Data-Driven Approaches , 2018, 2019 International Symposium on Medical Robotics (ISMR).

[16]  Michael C. Yip,et al.  Model-Less Feedback Control of Continuum Manipulators in Constrained Environments , 2014, IEEE Transactions on Robotics.

[17]  S. Misra,et al.  Three-Dimensional Needle Shape Reconstruction Using an Array of Fiber Bragg Grating Sensors , 2014, IEEE/ASME Transactions on Mechatronics.

[18]  Kaspar Althoefer,et al.  Highly dexterous 2‐module soft robot for intra‐organ navigation in minimally invasive surgery , 2018, The international journal of medical robotics + computer assisted surgery : MRCAS.

[19]  G. Passoni,et al.  Functional Evaluation of the Endotics System, a New Disposable Self-Propelled Robotic Colonoscope: in vitro tests and clinical trial , 2009, The International journal of artificial organs.

[20]  Kit-Hang Lee,et al.  Interfacing Soft and Hard: A Spring Reinforced Actuator. , 2020, Soft robotics.

[21]  Kit-Hang Lee,et al.  Vision-Based Online Learning Kinematic Control for Soft Robots Using Local Gaussian Process Regression , 2019, IEEE Robotics and Automation Letters.

[22]  Max Q.-H. Meng,et al.  Precise motion control of concentric-tube robot based on visual servoing , 2017, 2017 IEEE International Conference on Information and Automation (ICIA).