Contact-initiated shared control strategies for four-arm supernumerary manipulation with foot interfaces

In industrial or surgical settings, to achieve many tasks successfully, at least two people are needed. To this end, robotic assistance could be used to enable a single person to perform such tasks alone, with the help of robots through direct, shared, or autonomous control. We are interested in four-arm manipulation scenarios, where both feet are used to control two robotic arms via bi-pedal haptic interfaces. The robotic arms complement the tasks of the biological arms, for instance, in supporting and moving an object while working on it (using both hands). To reduce fatigue, cognitive workload, and to ease the execution of the foot manipulation, we propose two types of assistance that can be enabled upon contact with the object (i.e., based on the interaction forces): autonomous-contact force generation and auto-coordination of the robotic arms. The latter relates to controlling both arms with a single foot, once the object is grasped. We designed four (shared) control strategies that are derived from the combinations (absence/presence) of both assistance modalities, and we compared them through a user study (with 12 participants) on a four-arm manipulation task. The results show that force assistance positively improves human–robot fluency in the four-arm task, the ease of use and usefulness; it also reduces the fatigue. Finally, to make the dual-assistance approach the preferred and most successful among the proposed control strategies, delegating the grasping force to the robotic arms is a crucial factor when controlling them both with a single foot.

[1]  Thomas B. Sheridan,et al.  Space teleoperation through time delay: review and prognosis , 1993, IEEE Trans. Robotics Autom..

[2]  Paul A. Jennings,et al.  Haptic Foot Pedal: Influence of Shoe Type, Age, and Gender on Subjective Pulse Perception , 2018, Hum. Factors.

[3]  Mark Mulder,et al.  Neuromuscular Analysis as a Guideline in designing Shared Control , 2010 .

[4]  Aude Billard,et al.  Passive Interaction Control With Dynamical Systems , 2016, IEEE Robotics and Automation Letters.

[5]  Rebecca Brown,et al.  Evaluation of Subjective and EEG-Based Measures of Mental Workload , 2013, HCI.

[6]  Aude Billard,et al.  A Dynamical-System-Based Approach for Controlling Robotic Manipulators During Noncontact/Contact Transitions , 2018, IEEE Robotics and Automation Letters.

[7]  B. Argall,et al.  Human-in-the-Loop Optimization of Shared Autonomy in Assistive Robotics , 2017, IEEE Robotics and Automation Letters.

[8]  Manuel G. Catalano,et al.  Shared-Autonomy Control for Intuitive Bimanual Tele-Manipulation , 2018, 2018 IEEE-RAS 18th International Conference on Humanoid Robots (Humanoids).

[9]  Amir Shirkhodaie Supervised control of cooperative multi-agent robotic vehicles , 2002, Proceedings of the Thirty-Fourth Southeastern Symposium on System Theory (Cat. No.02EX540).

[10]  Bilge Mutlu,et al.  Shared control–based bimanual robot manipulation , 2019, Science Robotics.

[11]  Matthew Derry,et al.  Using machine learning to blend human and robot controls for assisted wheelchair navigation , 2013, 2013 IEEE 13th International Conference on Rehabilitation Robotics (ICORR).

[12]  John Kenneth Salisbury,et al.  The Intuitive/sup TM/ telesurgery system: overview and application , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[13]  I. Hunter,et al.  Dynamics of human ankle stiffness: variation with mean ankle torque. , 1982, Journal of biomechanics.

[14]  M Myrtek,et al.  Physical, mental, emotional, and subjective workload components in train drivers. , 1994, Ergonomics.

[15]  Marcello Bonfè,et al.  Bilateral teleoperation of a dual arms surgical robot with passive virtual fixtures generation , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[16]  Wilson Truccolo,et al.  Motor “Binding:” Do Functional Assemblies in Primary Motor Cortex Have a Role? , 2003, Neuron.

[17]  Frans C. T. van der Helm,et al.  A Task-Specific Analysis of the Benefit of Haptic Shared Control During Telemanipulation , 2013, IEEE Transactions on Haptics.

[18]  Anders Hansson,et al.  Semi-autonomous shared control of large-scale manipulator arms , 2010 .

[19]  Natalia Dounskaia,et al.  Egocentric and Allocentric Constraints in the Expression of Patterns of Interlimb Coordination , 1997, Journal of Cognitive Neuroscience.

[20]  Mark R. Cutkosky,et al.  Feedback Strategies for Telemanipulation with Shared Control of Object Handling Forces , 2005, Presence: Teleoperators & Virtual Environments.

[21]  Stephan P. Swinnen,et al.  Coordination constraints induced by effector combination under isofrequency and multifrequency conditions. , 1997 .

[22]  Long Cheng,et al.  Shared control for teleoperation enhanced by autonomous obstacle avoidance of robot manipulator , 2015, 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[23]  Martin Buss,et al.  A Survey of Environment-, Operator-, and Task-adapted Controllers for Teleoperation Systems , 2010 .

[24]  Stefano Stramigioli,et al.  Position Drift Compensation in Port-Hamiltonian Based Telemanipulation , 2006, 2006 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[25]  Dawn M. Tilbury,et al.  Blending of human and obstacle avoidance control for a high speed mobile robot , 2014, 2014 American Control Conference.

[26]  Sami Lini,et al.  Evaluating EEG Measures as a Workload Assessment in an Operational Video Game Setup , 2015, PhyCS.

[27]  Aude Billard,et al.  A Dynamical System Approach to Motion and Force Generation in Contact Tasks , 2019, Robotics: Science and Systems.

[28]  Fabio Babiloni,et al.  EEG-Based Mental Workload Neurometric to Evaluate the Impact of Different Traffic and Road Conditions in Real Driving Settings , 2018, Front. Hum. Neurosci..

[29]  Aude Billard,et al.  Avoidance of Convex and Concave Obstacles With Convergence Ensured Through Contraction , 2019, IEEE Robotics and Automation Letters.

[30]  Aude Billard,et al.  A Dynamical System Approach for Softly Catching a Flying Object: Theory and Experiment , 2016, IEEE Transactions on Robotics.

[31]  Lawrence W. Stark,et al.  A comparison of position and rate control for telemanipulations with consideration of manipulator system dynamics , 1987, IEEE Journal on Robotics and Automation.

[32]  Paolo Robuffo Giordano,et al.  A visual-based shared control architecture for remote telemanipulation , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[33]  V. Demberg,et al.  The Index of Cognitive Activity as a Measure of Cognitive Processing Load in Dual Task Settings , 2018, Front. Psychol..

[34]  Aude Billard,et al.  A dynamical system approach to task-adaptation in physical human–robot interaction , 2019, Auton. Robots.

[35]  Riccardo Muradore,et al.  An Energy Tank-Based Interactive Control Architecture for Autonomous and Teleoperated Robotic Surgery , 2015, IEEE Transactions on Robotics.

[36]  Laura Astolfi,et al.  Assessment of mental fatigue during car driving by using high resolution EEG activity and neurophysiologic indices , 2012, 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[37]  Guy Hoffman,et al.  Evaluating Fluency in Human–Robot Collaboration , 2019, IEEE Transactions on Human-Machine Systems.

[38]  Ruud G. J. Meulenbroek,et al.  Haptic feedback helps bipedal coordination , 2016, Experimental Brain Research.

[39]  Martial Hebert,et al.  Autonomy infused teleoperation with application to brain computer interface controlled manipulation , 2017, Autonomous Robots.

[40]  Katherine J. Kuchenbecker,et al.  Effects of Grip-Force, Contact, and Acceleration Feedback on a Teleoperated Pick-and-Place Task , 2017, IEEE Transactions on Haptics.

[41]  Satoshi Suzuki,et al.  Driving assistance through pedal reaction force control with consideration of JND , 2013, 2013 IEEE RO-MAN.

[42]  Frans C. T. van der Helm,et al.  Haptic Shared Control in Tele-Manipulation: Effects of Inaccuracies in Guidance on Task Execution , 2015, IEEE Transactions on Haptics.

[43]  Aude Billard,et al.  From Human Physical Interaction To Online Motion Adaptation Using Parameterized Dynamical Systems , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[44]  Yasuhisa Hasegawa,et al.  Modification of body schema by use of extra robotic thumb , 2018 .

[45]  Paolo Cavallari,et al.  Preferential coupling between voluntary movements of ipsilateral limbs , 1982, Neuroscience Letters.

[46]  Aude Billard,et al.  A dynamical system approach to realtime obstacle avoidance , 2012, Autonomous Robots.

[47]  Mark W. Spong,et al.  Bilateral teleoperation: An historical survey , 2006, Autom..

[48]  S. Swinnen,et al.  Two hands, one brain: cognitive neuroscience of bimanual skill , 2004, Trends in Cognitive Sciences.

[49]  Darwin G. Caldwell,et al.  Vision based virtual fixture generation for teleoperated robotic manipulation , 2016, 2016 International Conference on Advanced Robotics and Mechatronics (ICARM).

[50]  M. Buss,et al.  Transparent Exploration of Remote Environments by Internet Telepresence , 2003 .

[51]  Claudio Pacchierotti,et al.  Haptic-Based Shared-Control Methods for a Dual-Arm System , 2018, IEEE Robotics and Automation Letters.

[52]  Frans C. T. van der Helm,et al.  Force perception measurements at the foot , 2004, 2004 IEEE International Conference on Systems, Man and Cybernetics (IEEE Cat. No.04CH37583).

[53]  Stefano Stramigioli,et al.  Bilateral Telemanipulation With Time Delays: A Two-Layer Approach Combining Passivity and Transparency , 2011, IEEE Transactions on Robotics.

[54]  J. Kelso,et al.  Symmetry breaking dynamics of human multilimb coordination. , 1992, Journal of experimental psychology. Human perception and performance.

[55]  Siddhartha S. Srinivasa,et al.  A policy-blending formalism for shared control , 2013, Int. J. Robotics Res..

[56]  Markus H. Sneve,et al.  Pupil size signals mental effort deployed during multiple object tracking and predicts brain activity in the dorsal attention network and the locus coeruleus. , 2014, Journal of vision.

[57]  S. Hart,et al.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research , 1988 .

[58]  Claudio Pacchierotti,et al.  User Evaluation of a Haptic-Enabled Shared-Control Approach for Robotic Telemanipulation , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[59]  Ruud Beerens,et al.  Direct Force-Reflecting Two-Layer Approach for Passive Bilateral Teleoperation With Time Delays , 2018, IEEE Transactions on Robotics.

[60]  René van Paassen,et al.  Haptic guidance in bilateral teleoperation: Effects of guidance inaccuracy , 2015, 2015 IEEE World Haptics Conference (WHC).

[61]  T. Robinson,et al.  Minimally invasive surgery , 1999, European Surgical Research.

[62]  Dongjun Lee,et al.  Passive Bilateral Teleoperation With Constant Time Delay , 2006, IEEE Transactions on Robotics.

[63]  S. Siegler,et al.  The three-dimensional kinematics and flexibility characteristics of the human ankle and subtalar joints--Part I: Kinematics. , 1988, Journal of biomechanical engineering.

[64]  P.N. Brett,et al.  A surgical robot for cochleostomy , 2007, 2007 29th Annual International Conference of the IEEE Engineering in Medicine and Biology Society.

[65]  Hannes Bleuler,et al.  Four-Arm Manipulation via Feet Interfaces , 2019, ArXiv.

[66]  Thomas B. Sheridan,et al.  Telerobotics , 1989, Autom..

[67]  Diana Bohm,et al.  L2 Gain And Passivity Techniques In Nonlinear Control , 2016 .

[68]  R E Kearney,et al.  System identification of human ankle dynamics: intersubject variability and intrasubject reliability. , 1990, Clinical biomechanics.

[69]  H. Harry Asada,et al.  Design and control of Supernumerary Robotic Limbs for balance augmentation , 2015, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[70]  Louis B. Rosenberg,et al.  Virtual fixtures: Perceptual tools for telerobotic manipulation , 1993, Proceedings of IEEE Virtual Reality Annual International Symposium.