First-person view semi-autonomous teleoperation of cooperative wheeled mobile robots with visuo-haptic feedback

We propose a novel semi-autonomous teleoperation control framework for two nonholonomic wheeled mobile robots, which are cooperatively grasping and transporting a deformable object without any physical grip-enforcing fixture and are teleoperated by a single remote human user via a first-person view camera attached on the over-seeing wheeled mobile robot with some visuo-haptic feedback. First, we apply nonholonomic passive decomposition to split the kinematics of the two wheeled mobile robots into the grasping/first-person view-centering behavior and the teleoperation-related behavior. We reveal that, under a mixed constraint of the system (i.e. physical/nonholonomic no-slip/drift constraint and artificial/holonomic grasping/first-person view-centering constraint), the first-person view camera cannot track an arbitrary human command. We then design the semi-autonomous control architecture, consisting of an autonomous grasping-enforcing and first-person view-centering control and a first-person view camera teleoperation control, which in turns stands upon autonomous human command tracking control. We also design visual and haptic feedback to notify the user of, and also nudge them to reduce, the command-behavior mismatch caused by the mixed constraint, which is difficult for the user to understand via the limited first-person view, we can thus induce user confusion/frustration and consequently performance deterioration. Experiment verification and a human subject study are performed to show the efficacy of our proposed first-person view semi-autonomous teleoperation framework with the visuo-haptic feedback.