A Fuzzy Reinforcement Learning Approach for Continuum Robot Control

Continuum robots (CRs) hold great potential for many medical and industrial applications where compliant interaction within the potentially confined environment is required. However, the navigation of CRs poses several challenges due to their limited actuation channels and the hyper-flexibility of their structure. Environmental uncertainty and characteristic hysteresis in such procedures add to the complexity of their operation. Therefore, the quality of trajectory tracking for continuum robots plays an essential role in the success of the application procedures. While there are a few different actuation configurations available for CRs, the focus of this paper will be placed on tendon-driven manipulators. In this research, a new fuzzy reinforcement learning (FRL) approach is introduced. The proposed FRL-based control parameters are tuned by the Taguchi method and evolutionary genetic algorithm (GA) to provide faster convergence to the Nash Equilibrium. The approach is verified through a comprehensive set of simulations using a Cosserat rod model. The results show a steady and accurate trajectory tracking capability for a CR.

[1]  Pietro Valdastri,et al.  A disposable continuum endoscope using piston-driven parallel bellow actuator , 2018, 2018 International Symposium on Medical Robotics (ISMR).

[2]  Lei Wang,et al.  Towards Characterization and Adaptive Compensation of Backlash in a Novel Robotic Catheter System for Cardiovascular Interventions , 2018, IEEE Transactions on Biomedical Circuits and Systems.

[3]  Chao Liu,et al.  Task-space position control of concentric-tube robot with inaccurate kinematics using approximate Jacobian , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[4]  Farrokh Janabi-Sharifi,et al.  Cooperative Continuum Robots: Concept, Modeling, and Workspace Analysis , 2018, IEEE Robotics and Automation Letters.

[5]  Hongyi Li,et al.  Adaptive Fuzzy PID Controller of a Master-Slave Robotic Catheter System in Minimally Invasive Surgery , 2013 .

[6]  Derong Liu,et al.  Integral Reinforcement Learning for Linear Continuous-Time Zero-Sum Games With Completely Unknown Dynamics , 2014, IEEE Transactions on Automation Science and Engineering.

[7]  Wenfu Xu,et al.  A comprehensive static model of cable-driven multi-section continuum robots considering friction effect , 2019, Mechanism and Machine Theory.

[8]  Aparajita Sengupta,et al.  Control of Single-Segment Continuum Robots: Reinforcement Learning vs. Neural Network based PID , 2018, 2018 International Conference on Control, Power, Communication and Computing Technologies (ICCPCCT).

[9]  Michael P. Wellman,et al.  Nash Q-Learning for General-Sum Stochastic Games , 2003, J. Mach. Learn. Res..

[10]  Girish Chowdhary,et al.  Open Loop Position Control of Soft Continuum Arm Using Deep Reinforcement Learning , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[11]  Michael C. Yip,et al.  Model-Less Feedback Control of Continuum Manipulators in Constrained Environments , 2014, IEEE Transactions on Robotics.

[12]  Emmanuel Vander Poorten,et al.  Towards Autonomous Robotic Catheter Navigation Using Reinforcement Learning , 2014 .

[13]  Richard S. Sutton,et al.  Generalization in ReinforcementLearning : Successful Examples UsingSparse Coarse , 1996 .

[14]  Natalia Akchurina,et al.  Multiagent reinforcement learning: algorithm converging to Nash equilibrium in general-sum discounted stochastic games , 2009, AAMAS.

[15]  John Kenneth Salisbury,et al.  Configuration Tracking for Continuum Manipulators With Coupled Tendon Drive , 2009, IEEE Transactions on Robotics.

[16]  Nabil Simaan,et al.  Compliant Motion Control for Multisegment Continuum Robots With Actuation Force Sensing , 2014, IEEE Transactions on Robotics.

[17]  Richard S. Sutton,et al.  Reinforcement Learning is Direct Adaptive Optimal Control , 1992, 1991 American Control Conference.

[18]  Heikki Handroos,et al.  ANFIS and Fuzzy Tuning of PID Controller for Trajectory Tracking of a Flexible Hydraulically Driven Parallel Robot Machine , 2013 .

[19]  Michael C. Yip,et al.  Model-Less Hybrid Position/Force Control: A Minimalist Approach for Continuum Manipulators in Unknown, Constrained Environments , 2016, IEEE Robotics and Automation Letters.

[20]  Michael C. Yip,et al.  Autonomous Control of Continuum Robot Manipulators for Complex Cardiac Ablation Tasks , 2017, J. Medical Robotics Res..

[21]  Ian D. Walker,et al.  Task-space control of extensible continuum manipulators , 2011, 2011 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[22]  Jessica Burgner-Kahrs,et al.  Toward Motion Coordination Control and Design Optimization for Dual-Arm Concentric Tube Continuum Robots , 2018, IEEE Robotics and Automation Letters.

[23]  Cecilia Laschi,et al.  Model-Based Reinforcement Learning for Closed-Loop Dynamic Control of Soft Robotic Manipulators , 2019, IEEE Transactions on Robotics.

[24]  Mircea Ivanescu,et al.  A variable structure controller for a tentacle manipulator , 1995, Proceedings of 1995 IEEE International Conference on Robotics and Automation.

[25]  Masoud Goharimanesh,et al.  Yaw Moment Control Using Fuzzy Reinforcemnt Learning , 2014 .

[26]  Farrokh Janabi-Sharifi,et al.  Model-Based Force Estimation for Intracardiac Catheters , 2016, IEEE/ASME Transactions on Mechatronics.

[27]  Masoud Goharimanesh,et al.  Diabetic Control Using Genetic Fuzzy-PI Controller , 2014 .

[28]  Farrokh Janabi-Sharifi,et al.  Robot-assisted catheter manipulation for intracardiac navigation , 2009, International Journal of Computer Assisted Radiology and Surgery.

[29]  Lotfi A. Zadeh,et al.  Toward extended fuzzy logic - A first step , 2009, Fuzzy Sets Syst..

[30]  Kaspar Althoefer,et al.  Kinematic Control of Continuum Manipulators Using a Fuzzy-Model-Based Approach , 2016, IEEE Transactions on Industrial Electronics.

[31]  Gang Yu,et al.  3D Path Planning for Anterior Spinal Surgery Based on CT images and Reinforcement Learning* , 2018, 2018 IEEE International Conference on Cyborg and Bionic Systems (CBS).

[32]  Jinwoo Jung,et al.  An evaluation of closed-loop control options for continuum manipulators , 2012, 2012 IEEE International Conference on Robotics and Automation.

[33]  Hao Jiang,et al.  Model-free control for soft manipulators based on reinforcement learning , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[34]  Farrokh Janabi-Sharifi,et al.  An Efficient Static Analysis of Continuum Robots , 2014 .

[35]  Bart De Schutter,et al.  A Comprehensive Survey of Multiagent Reinforcement Learning , 2008, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[36]  Jaesoon Choi,et al.  Automatic control of cardiac ablation catheter with deep reinforcement learning method , 2019, Journal of Mechanical Science and Technology.

[37]  Wei Zhang,et al.  A Unified Framework for Street-View Panorama Stitching , 2016, Sensors.

[38]  Xiao Chen,et al.  Laparoscope arm automatic positioning for robot-assisted surgery based on reinforcement learning , 2019, Mechanical Sciences.

[39]  Wei Wang,et al.  Variable Admittance Control Based on Fuzzy Reinforcement Learning for Minimally Invasive Surgery Manipulator , 2017, Sensors.

[40]  Lotfi A. Zadeh,et al.  Is there a need for fuzzy logic? , 2008, NAFIPS 2008 - 2008 Annual Meeting of the North American Fuzzy Information Processing Society.

[41]  Ian D. Walker,et al.  Large deflection dynamics and control for planar continuum robots , 2001 .

[42]  Mohammad-Bagher Naghibi-Sistani,et al.  An intelligent controller for ionic polymer metal composites using optimized fuzzy reinforcement learning , 2017, J. Intell. Fuzzy Syst..

[43]  Christian Duriez,et al.  Motion Control of Cable-Driven Continuum Catheter Robot Through Contacts , 2019, IEEE Robotics and Automation Letters.

[44]  Jinwoo Jung,et al.  Towards closed loop control of a continuum robotic manipulator for medical applications , 2011, 2011 IEEE International Conference on Robotics and Automation.

[45]  Amit Konar,et al.  An Efficient Computing of Correlated Equilibrium for Cooperative $Q$ -Learning-Based Multi-Robot Planning , 2020, IEEE Transactions on Systems, Man, and Cybernetics: Systems.

[46]  D.M. Dawson,et al.  Neural Network Grasping Controller for Continuum Robots , 2006, Proceedings of the 45th IEEE Conference on Decision and Control.

[47]  Dong-Soo Kwon,et al.  Path Planning for Automation of Surgery Robot based on Probabilistic Roadmap and Reinforcement Learning , 2018, 2018 15th International Conference on Ubiquitous Robots (UR).

[48]  Masoud Goharimanesh,et al.  Optimum parameters of nonlinear integrator using design of experiments based on Taguchi method , 2015 .

[49]  Sohrab Khanmohammadi,et al.  A soft robotics nonlinear hybrid position/force control for tendon driven catheters , 2017 .

[50]  Masoud Goharimanesh,et al.  COMBINING THE PRINCIPLES OF FUZZY LOGIC AND REINFORCEMENT LEARNING FOR CONTROL OF DYNAMIC SYSTEMS , 2015 .

[51]  Cagdas D. Onal,et al.  Design and control of a soft and continuously deformable 2D robotic manipulation system , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[52]  Virginia Torczon,et al.  On the Convergence of Pattern Search Algorithms , 1997, SIAM J. Optim..

[53]  Oliver Sawodny,et al.  Model-based feedforward position control of constant curvature continuum robots using feedback linearization , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[54]  Jian S. Dai,et al.  Model-Free Control for Continuum Robots Based on an Adaptive Kalman Filter , 2018, IEEE/ASME Transactions on Mechatronics.

[55]  Peter Dayan,et al.  Technical Note: Q-Learning , 2004, Machine Learning.