Autonomy infused teleoperation with application to brain computer interface controlled manipulation

Robot teleoperation systems face a common set of challenges including latency, low-dimensional user commands, and asymmetric control inputs. User control with Brain–Computer Interfaces (BCIs) exacerbates these problems through especially noisy and erratic low-dimensional motion commands due to the difficulty in decoding neural activity. We introduce a general framework to address these challenges through a combination of computer vision, user intent inference, and arbitration between the human input and autonomous control schemes. Adjustable levels of assistance allow the system to balance the operators capabilities and their perception of control authority. Additionally, a custom servo controller design allow for safe interactions of the robotic arm with the environment. We present experimental results demonstrating significant performance improvement using our shared-control assistance framework on adapted rehabilitation benchmarks with two subjects at various timepoints relative to their implantation with intracortical BCIs. Our results indicate that shared assistance mitigates perceived user difficulty in using a seven-degree of freedom robotic arm as a prosthetic and enables successful performance on previously infeasible tasks. We showcase the extensibility of our architecture with applications to quality-of-life tasks such as opening a door, pouring liquids from containers, and manipulation with objects previously unknown to the system in densely cluttered environments.