Comparative Assessment of Sensing Modalities on Manipulation Failure Detection

Execution monitoring is important for the robot to safely interact with its environment, and to successfully complete the given tasks. This is because several unexpected outcomes that may occur during manipulation in unstructured environments (i.e., in homes) such as sensory noises, improper action parameters, hardware limitations or external factors. The execution monitoring process should be continuous for effective failure detection and prevention if possible. We present an empirical analysis of proprioception, audition and vision modalities to detect failures on a selected tabletop object manipulation actions. We model failure detection as a binary classification problem, where the classifier uses high level predicates extracted from raw sensory measurements. We evaluate the contributions of these modalities in detecting failures for pick, place and push actions on a Baxter robot.