DexVIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video

Dexterous multi-fingered robotic hands have a formidable action space, 1 yet their morphological similarity to the human hand holds immense potential to 2 accelerate robot learning. We propose DexVIP, an approach to learn dexterous 3 robotic grasping from human-object interactions present in in-the-wild YouTube 4 videos. We do this by curating grasp images from human-object interaction videos 5 and imposing a prior over the agent’s hand pose when learning to grasp with deep 6 reinforcement learning. A key advantage of our method is that the learned policy 7 is able to leverage free-form in-the-wild visual data. As a result, it can easily 8 scale to new objects, and it sidesteps the standard practice of collecting human 9 demonstrations in a lab—a much more expensive and indirect way to capture 10 human expertise. Through experiments on 27 objects with a 30-DoF simulated 11 robot hand, we demonstrate that DexVIP compares favorably to existing approaches 12 that lack a hand pose prior or rely on specialized tele-operation equipment to obtain 13 human demonstrations, while also being faster to train. 14