A Systematic Review of Multilateral Teleoperation Systems

While conventional bilateral Single-Master/Single-Slave (SM/SS) teleoperation systems have received considerable attention during the past several decades, multilateral teleoperation is only recently being studied. Unlike an SM/SS system, which consists of one master-slave set, multilateral teleoperation frameworks involve a minimum of three agents in order to remotely perform a task. This paper presents an overview of multilateral teleoperation systems and classifies the existing state-of-the-art architectures based on topologies, applications, and closed-loop stability analysis. For each category, the review discusses control strategies used for various architectures as well as control challenges (e.g., closed-loop instability as a result of a delay in the communication network) for each methodology.

[1]  Shahin Sirouspour,et al.  Trilateral teleoperation control of kinematically redundant robotic manipulators , 2011, Int. J. Robotics Res..

[2]  Kouhei Ohnishi,et al.  Multi-robot teleoperation under dynamically changing network topology , 2009, 2009 IEEE International Conference on Industrial Technology.

[3]  Mahdi Tavakoli,et al.  Passivity and Absolute Stability Analysesof Trilateral Haptic Collaborative Systems , 2015, J. Intell. Robotic Syst..

[4]  Jee-Hwan Ryu,et al.  Multilateral control for delayed teleoperation , 2013, 2013 16th International Conference on Advanced Robotics (ICAR).

[5]  Hideki Hashimoto,et al.  Development of a Human-Robot-Shared Controlled Teletweezing System , 2007, IEEE Transactions on Control Systems Technology.

[6]  Di-Hua Zhai,et al.  Multilateral Telecoordinated Control of Multiple Robots With Uncertain Kinematics , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[7]  B. Hannaford,et al.  Virtual Coupling Schemes for Position Coherency in Networked Haptic Environments , 2006, The First IEEE/RAS-EMBS International Conference on Biomedical Robotics and Biomechatronics, 2006. BioRob 2006..

[8]  Yuanqing Xia,et al.  Adaptive Fuzzy Control of Multilateral Asymmetric Teleoperation for Coordinated Multiple Mobile Manipulators , 2016, IEEE Transactions on Fuzzy Systems.

[9]  Rajnikant V. Patel,et al.  An Internet-based distributed multiple-telerobot system , 2003, IEEE Trans. Syst. Man Cybern. Part A.

[10]  Kunikatsu Takase,et al.  Multioperator Teleoperation of Multirobot Systems with Time Delay: Part IAids for Collision-Free Control , 2002, Presence: Teleoperators & Virtual Environments.

[11]  Peter M. Vishton,et al.  Haptically Linked Dyads , 2006, Psychological science.

[12]  Rajnikant V. Patel,et al.  A dual-user teleoperated system with Virtual Fixtures for robotic surgical training , 2013, 2013 IEEE International Conference on Robotics and Automation.

[13]  H. A. Talebi,et al.  A new set of desired objectives for dual-user systems in the presence of unknown communication delay , 2011, 2011 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM).

[14]  Keyvan Hashtrudi-Zaad,et al.  Performance Issues in Collaborative Haptic Training , 2007, Proceedings 2007 IEEE International Conference on Robotics and Automation.

[15]  Fazel Naghdy,et al.  Stability Control of Force-Reflected Nonlinear Multilateral Teleoperation System under Time-Varying Delays , 2016, J. Sensors.

[16]  Nobuto Matsuhira,et al.  A collaborative multi-site teleoperation over an ISDN , 2003 .

[17]  Blake Hannaford,et al.  Architectures for shared haptic virtual environments , 1997, Comput. Graph..

[18]  Panfeng Huang,et al.  Auxiliary asymmetric dual-user shared control method for teleoperation , 2015, 2015 12th International Conference on Ubiquitous Robots and Ambient Intelligence (URAI).

[19]  Paul G. Backes,et al.  Supervised autonomous control, shared control, and teleoperation for space servicing , 1993 .

[20]  Mahdi Tavakoli,et al.  Stability of cooperative teleoperation using haptic devices with complementary degrees of fr , 2014 .

[21]  Keyvan Hashtrudi-Zaad,et al.  Analysis of Coupled Stability in Multilateral Dual-User Teleoperation Systems , 2014, IEEE Transactions on Robotics.

[22]  Kenneth Y. Goldberg,et al.  Collaborative teleoperation via the Internet , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[23]  Dongjun Lee,et al.  Bilateral Teleoperation of Multiple Cooperative Robots over Delayed Communication Networks: Theory , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[24]  Shahin Sirouspour,et al.  Multi-operator/multi-robot teleoperation: an adaptive nonlinear control approach , 2005, 2005 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Hiroo Iwata,et al.  Cooperative Work in Virtual Environment with Force Feed-back , 1995 .

[26]  Daniela Constantinescu,et al.  Distributed haptic cooperation with passive multirate wave communications , 2012, 2012 IEEE Haptics Symposium (HAPTICS).

[27]  Hugo Jiménez-Hernández,et al.  Higher order sliding mode based impedance control for dual-user bilateral teleoperation under unknown constant time delay , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[28]  Zhi Li.  Networked haptic cooperation using remote dynamic proxies , 2008 .

[29]  Mary C. Whitton,et al.  Effective Cooperative Haptic Interaction over the Internet , 2007, 2007 IEEE Virtual Reality Conference.

[30]  Guang-Ren Duan,et al.  Trilateral Teleoperation of Adaptive Fuzzy Force/Motion Control for Nonlinear Teleoperators With Communication Random Delays , 2013, IEEE Transactions on Fuzzy Systems.

[31]  M. Kawato,et al.  Two is better than one: Physical interactions improve motor performance in humans , 2014, Scientific Reports.

[32]  Blake Hannaford,et al.  Experimental comparison of internet haptic collaboration with time-delay compensation techniques , 2008, 2008 IEEE International Conference on Robotics and Automation.

[33]  Tanneguy Redarce,et al.  A dual-user teleoperation system with Online Authority Adjustment for haptic training , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[34]  Lawrence H. Kim,et al.  Design and evaluation of a trilateral shared-control architecture for teleoperated training robots , 2015, 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC).

[35]  Rajnikant V. Patel,et al.  A framework for supervised robotics-assisted mirror rehabilitation therapy , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[36]  Heidar Ali Talebi,et al.  An expertise-oriented training framework for robotics-assisted surgery , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[37]  Yuanqing Xia,et al.  Adaptive Fuzzy Control for Multilateral Cooperative Teleoperation of Multiple Robotic Manipulators Under Random Network-Induced Delays , 2014, IEEE Transactions on Fuzzy Systems.

[38]  Martin Buss,et al.  Model-Mediated Teleoperation for multi-operator multi-robot systems , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[39]  B. Khademian,et al.  A robust multilateral shared controller for dual-user teleoperation systems , 2008, 2008 Canadian Conference on Electrical and Computer Engineering.

[40]  Shahin Sirouspour,et al.  Stability and Performance Analysis of Centralized and Distributed Multi-rate Control Architectures for Multi-user Haptic Interaction , 2007, Int. J. Robotics Res..

[41]  Heidar Ali Talebi,et al.  Networked dual-user teleoperation with time-varying authority adjustment: A wave variable approach , 2014, 2014 IEEE/ASME International Conference on Advanced Intelligent Mechatronics.

[42]  Mark W. Noakes,et al.  Large scale multi-fingered end-effector teleoperation , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[43]  A Ghorbanian,et al.  A novel control framework for nonlinear time-delayed dual-master/single-slave teleoperation. , 2013, ISA transactions.

[44]  Yasuyoshi Yokokohji,et al.  Multilateral teleoperation control over time-delayed computer networks using wave variables , 2012, 2012 IEEE Haptics Symposium (HAPTICS).

[45]  Mahdi Tavakoli,et al.  Absolute Stability of Multi-DOF Multilateral Haptic Systems , 2014, IEEE Transactions on Control Systems Technology.

[46]  Rajni V. Patel,et al.  Novel Cooperative Teleoperation Framework: Multi-Master/Single-Slave System , 2015, IEEE/ASME Transactions on Mechatronics.

[47]  Mark R. Cutkosky,et al.  Feedback strategies for shared control in dexterous telemanipulation , 2003, Proceedings 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003) (Cat. No.03CH37453).

[48]  Martin Buss,et al.  Development of a Multi-modal Multi-user Telepresence and Teleaction System , 2010, Int. J. Robotics Res..

[49]  Daniela Constantinescu,et al.  Comparison of power- and wave-based control of remote dynamic proxies for networked haptic cooperation , 2009, 2009 International Conference on Mechatronics and Automation.

[50]  Robert Richardson,et al.  A Control Strategy for Upper Limb Robotic Rehabilitation With a Dual Robot System , 2010, IEEE/ASME Transactions on Mechatronics.

[51]  Claudio Melchiorri,et al.  A Performance and Stability Analysis for Cooperative Teleoperation Systems , 2011 .

[52]  Allison M. Okamura,et al.  Two is not always better than one: Effects of teleoperation and haptic coupling , 2016, 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[53]  Kyle B. Reed,et al.  Haptic cooperation between people, and between people and machines , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[54]  Keyvan Hashtrudi-Zaad,et al.  Unconditional stability analysis of dual-user teleoperation systems , 2010, 2010 IEEE Haptics Symposium.

[55]  Martin Buss,et al.  Shared-Control Paradigms in Multi-Operator-Single-Robot Teleoperation , 2009, Human Centered Robot Systems, Cognition, Interaction, Technology.

[56]  Shahin Sirouspour,et al.  Cooperative teleoperation control with projective force mappings , 2010, 2010 IEEE Haptics Symposium.

[57]  Marcia Kilchenman O'Malley,et al.  Improving the retention of motor skills after reward-based reinforcement by incorporating haptic guidance and error augmentation , 2016, 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics (BioRob).

[58]  Robert L. Williams,et al.  Shared control of multiple-manipulator, sensor-based telerobotic systems , 1997, Proceedings of International Conference on Robotics and Automation.

[59]  Keyvan Hashtrudi-Zaad,et al.  Dual-User Teleoperation Systems: New Multilateral Shared Control Architecture and Kinesthetic Performance Measures , 2012, IEEE/ASME Transactions on Mechatronics.

[60]  Paul Richard,et al.  Haptic guides in cooperative virtual environments: Design and human performance evaluation , 2010, 2010 IEEE Haptics Symposium.

[61]  Ranjan Mukherjee,et al.  A shared-control approach to haptic interface design for minimally invasive telesurgical training , 2005, IEEE Transactions on Control Systems Technology.

[62]  Shahin Sirouspour,et al.  Haptic-enabled Collaborative Training with Generalized Force and Position Mappings , 2008, 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[63]  Jee-Hwan Ryu,et al.  Stable multilateral teleoperation with Time Domain Passivity Approach , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[64]  Alin Albu-Schäffer,et al.  A Peer-to-Peer Trilateral Passivity Control for Delayed Collaborative Teleoperation , 2012, EuroHaptics.

[65]  Craig R. Carignan,et al.  Cooperative control of virtual objects over the Internet using force-reflecting master arms , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[66]  Allison M. Okamura,et al.  A paced shared-control teleoperated architecture for supervised automation of multilateral surgical tasks , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[67]  Samad Hayati,et al.  Design and implementation of a robot control system with traded and shared control capability , 1989, Proceedings, 1989 International Conference on Robotics and Automation.

[68]  Blake Hannaford,et al.  Experimental Internet Haptic Collaboration using Virtual Coupling Schemes , 2008, 2008 Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[69]  Mahdi Tavakoli,et al.  Therapist-in-the-Loop robotics-assisted mirror rehabilitation therapy: An Assist-as-Needed framework , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[70]  Alicia P Melis,et al.  The evolutionary roots of human collaboration: coordination and sharing of resources , 2013, Annals of the New York Academy of Sciences.

[71]  Ya-Jun Pan,et al.  Integrated adaptive robust control for multilateral teleoperation systems under arbitrary time delays , 2016 .

[72]  Ning Xi,et al.  Cooperative teleoperation of a multirobot system with force reflection via Internet , 2004, IEEE/ASME Transactions on Mechatronics.

[73]  Dongjun Lee,et al.  Bilateral Teleoperation of Multiple Cooperative Robots over Delayed Communication Networks: Application , 2005, Proceedings of the 2005 IEEE International Conference on Robotics and Automation.

[74]  Kam S. Tso,et al.  UMI: an interactive supervisory and shared control system for telerobotics , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[75]  ChangSu Ha,et al.  Semiautonomous Haptic Teleoperation Control Architecture of Multiple Unmanned Aerial Vehicles , 2013, IEEE/ASME Transactions on Mechatronics.

[76]  Matt Adcock,et al.  Using collaborative haptics in remote surgical training , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[77]  Keyvan Hashtrudi-Zaad,et al.  Shared control architectures for haptic training: Performance and coupled stability analysis , 2011, Int. J. Robotics Res..

[78]  Shahin Sirouspour,et al.  Modeling and control of cooperative teleoperation systems , 2005, IEEE Transactions on Robotics.

[79]  Ali Shahdi,et al.  Discrete-time multi-model control for cooperative teleoperation under time delay , 2006, Proceedings 2006 IEEE International Conference on Robotics and Automation, 2006. ICRA 2006..

[80]  Marcia Kilchenman O'Malley,et al.  Negative efficacy of fixed gain error reducing shared control for training in virtual environments , 2009, TAP.

[81]  Daniela Constantinescu,et al.  Remote dynamic proxies for wave-based peer-to-peer haptic interaction , 2009, World Haptics 2009 - Third Joint EuroHaptics conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems.

[82]  Kazuhiro Kosuge,et al.  Control of single-master multi-slave manipulator system using VIM , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[83]  H. A. Talebi,et al.  A robust control architecture for dual user teleoperation system with time-delay , 2010, IECON 2010 - 36th Annual Conference on IEEE Industrial Electronics Society.

[84]  Anuradha M. Annaswamy,et al.  Motion synchronization in virtual environments with shared haptics and large time delays , 2005, First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference.

[85]  Keyvan Hashtrudi-Zaad,et al.  A Framework for Unconditional Stability Analysis of Multimaster/Multislave Teleoperation Systems , 2013, IEEE Transactions on Robotics.

[86]  Shahin Sirouspour,et al.  A Kinematic Control Framework for Single-Slave Asymmetric Teleoperation Systems , 2011, IEEE Transactions on Robotics.

[87]  Mark W. Spong,et al.  Bilateral teleoperation: An historical survey , 2006, Autom..

[88]  Abdelouahab Zaatri,et al.  Investigations in telerobotics using cooperative supervisory modes of control , 1997, Other Conferences.

[89]  Fazel Naghdy,et al.  Enhancing flexibility of the dual-master-dual-slave multilateral teleoperation system , 2015, 2015 IEEE Conference on Control Applications (CCA).

[90]  Thomas B. Sheridan,et al.  Telerobotics , 1989, Autom..

[91]  Keyvan Hashtrudi-Zaad,et al.  Development of a Guaranteed Stable Network of Telerobots with Kinesthetic Consensus , 2014, IEEE Transactions on Haptics.

[92]  Dongjun Lee,et al.  Peer-to-peer control architecture for multiuser haptic collaboration over undirected delayed packet-switching network , 2010, 2010 IEEE International Conference on Robotics and Automation.

[93]  Mahdi Tavakoli,et al.  A method for passivity analysis of multilateral haptic systems , 2014, Adv. Robotics.

[94]  S. Hayati,et al.  A testbed for a unified teleoperated-autonomous dual-arm robotic system , 1990, Proceedings., IEEE International Conference on Robotics and Automation.

[95]  Mahdi Tavakoli,et al.  Robotics-Assisted Mirror Rehabilitation Therapy: A Therapist-in-the-Loop Assist-as-Needed Architecture , 2016, IEEE/ASME Transactions on Mechatronics.

[96]  Yuanqing Xia,et al.  Neural Network-Based Control of Networked Trilateral Teleoperation With Geometrically Unknown Constraints , 2016, IEEE Transactions on Cybernetics.

[97]  Kouhei Ohnishi,et al.  A realization of haptic training system by multilateral control , 2004, 30th Annual Conference of IEEE Industrial Electronics Society, 2004. IECON 2004.

[98]  Martin Buss,et al.  A Survey of Environment-, Operator-, and Task-adapted Controllers for Teleoperation Systems , 2010 .

[99]  Martin Buss,et al.  Efficiency analysis in a collaborative task with reciprocal haptic feedback , 2009, 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[100]  Daniela Constantinescu,et al.  Centralized multi-user multi-rate haptic cooperation using wave transformation , 2009, 2009 International Conference on Mechatronics and Automation.

[101]  Kiyoshi Ohishi,et al.  A Realization of Multilateral Force Feedback Control for Cooperative Motion , 2007, 2006 IEEE International Conference on Industrial Technology.

[102]  Marcia Kilchenman O'Malley,et al.  The Task-Dependent Efficacy of Shared-Control Haptic Guidance Paradigms , 2012, IEEE Transactions on Haptics.

[103]  P.X. Liu,et al.  Design and implementation of a collaborative virtual haptic surgical training system , 2005, IEEE International Conference Mechatronics and Automation, 2005.

[104]  Martin Buss,et al.  Enhancement of Multi-user Teleoperation Systems by Prediction of Dyadic Haptic Interaction , 2010, ISER.

[105]  Mohammad Javad Yazdanpanah,et al.  A sliding-mode controller for dual-user teleoperation with unknown constant time delays , 2012, Robotica.