Language-Conditioned Path Planning

Contact is at the core of robotic manipulation. At times, it is desired (e.g. manipulation and grasping), and at times, it is harmful (e.g. when avoiding obstacles). However, traditional path planning algorithms focus solely on collision-free paths, limiting their applicability in contact-rich tasks. To address this limitation, we propose the domain of Language-Conditioned Path Planning, where contact-awareness is incorporated into the path planning problem. As a first step in this domain, we propose Language-Conditioned Collision Functions (LACO) a novel approach that learns a collision function using only a single-view image, language prompt, and robot configuration. LACO predicts collisions between the robot and the environment, enabling flexible, conditional path planning without the need for manual object annotations, point cloud data, or ground-truth object meshes. In both simulation and the real world, we demonstrate that LACO can facilitate complex, nuanced path plans that allow for interaction with objects that are safe to collide, rather than prohibiting any collision.

[1]  P. Abbeel,et al.  Language Reward Modulation for Pretraining Reinforcement Learning , 2023, ArXiv.

[2]  Beomjoon Kim,et al.  Local object crop collision network for efficient simulation of non-convex objects in GPU-based simulators , 2023, Robotics: Science and Systems.

[3]  D. Fox,et al.  CabiNet: Scaling Neural Collision Detection for Object Rearrangement with Procedural Scene Generation , 2023, 2023 IEEE International Conference on Robotics and Automation (ICRA).

[4]  P. Abbeel,et al.  Multi-View Masked World Models for Visual Robotic Manipulation , 2023, ICML.

[5]  Ali Farhadi,et al.  Objaverse: A Universe of Annotated 3D Objects , 2022, 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Pannag R. Sanketi,et al.  RT-1: Robotics Transformer for Real-World Control at Scale , 2022, Robotics: Science and Systems.

[7]  P. Abbeel,et al.  Sim-to-Real via Sim-to-Seg: End-to-end Off-road Autonomous Driving Without Real Data , 2022, CoRL.

[8]  P. Abbeel,et al.  Instruction-Following Agents with Multimodal Transformer , 2022, 2210.13431.

[9]  Peter R. Florence,et al.  Code as Policies: Language Model Programs for Embodied Control , 2022, 2023 IEEE International Conference on Robotics and Automation (ICRA).

[10]  S. Levine,et al.  Do As I Can, Not As I Say: Grounding Language in Robotic Affordances , 2022, CoRL.

[11]  A. Zamir,et al.  MultiMAE: Multi-modal Multi-task Masked Autoencoders , 2022, ECCV.

[12]  Sergey Levine,et al.  BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning , 2022, CoRL.

[13]  Dongdong Chen,et al.  CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields , 2021, 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Dieter Fox,et al.  CLIPort: What and Where Pathways for Robotic Manipulation , 2021, CoRL.

[15]  Ilya Sutskever,et al.  Learning Transferable Visual Models From Natural Language Supervision , 2021, ICML.

[16]  George J. Pappas,et al.  Perception-Based Temporal Logic Planning in Uncertain Semantic Maps , 2020, IEEE Transactions on Robotics.

[17]  Yiqing Liang,et al.  SSCNav: Confidence-Aware Semantic Scene Completion for Visual Semantic Navigation , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[18]  Dieter Fox,et al.  Object Rearrangement Using Learned Implicit Collision Functions , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[19]  Shuran Song,et al.  Learning a Decentralized Multi-arm Motion Planner , 2020, CoRL.

[20]  Sean L. Bowman,et al.  Reactive Semantic Planning in Unexplored Semantic Environments Using Deep Perceptual Feedback , 2020, IEEE Robotics and Automation Letters.

[21]  Michael C. Yip,et al.  Motion Planning Networks: Bridging the Gap Between Learning-Based and Classical Motion Planners , 2019, IEEE Transactions on Robotics.

[22]  Andrew J. Davison,et al.  PyRep: Bringing V-REP to Deep Robot Learning , 2019, ArXiv.

[23]  Sergey Levine,et al.  Sim-To-Real via Sim-To-Sim: Data-Efficient Robotic Grasping via Randomized-To-Canonical Adaptation Networks , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Andrew J. Davison,et al.  Sim-to-Real Reinforcement Learning for Deformable Object Manipulation , 2018, CoRL.

[25]  Michael C. Yip,et al.  Motion Planning Networks , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[26]  Andrew J. Davison,et al.  Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task , 2017, CoRL.

[27]  Leonidas J. Guibas,et al.  ShapeNet: An Information-Rich 3D Model Repository , 2015, ArXiv.

[28]  Siddhartha S. Srinivasa,et al.  The YCB object and Model set: Towards common benchmarks for manipulation research , 2015, 2015 International Conference on Advanced Robotics (ICAR).

[29]  Pieter Abbeel,et al.  Combined task and motion planning through an extensible planner-independent interface layer , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[30]  Pieter Abbeel,et al.  Finding Locally Optimal, Collision-Free Trajectories with Sequential Convex Optimization , 2013, Robotics: Science and Systems.

[31]  Maren Bennewitz,et al.  Whole-body motion planning for manipulation of articulated objects , 2013, 2013 IEEE International Conference on Robotics and Automation.

[32]  Emilio Frazzoli,et al.  Sampling-based algorithms for optimal motion planning , 2011, Int. J. Robotics Res..

[33]  P. Cochat,et al.  Et al , 2008, Archives de pediatrie : organe officiel de la Societe francaise de pediatrie.

[34]  Steven M. LaValle,et al.  RRT-connect: An efficient approach to single-query path planning , 2000, Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No.00CH37065).

[35]  Nancy M. Amato,et al.  A randomized roadmap method for path and manipulation planning , 1996, Proceedings of IEEE International Conference on Robotics and Automation.

[36]  Lydia E. Kavraki,et al.  Randomized preprocessing of configuration for fast path planning , 1994, Proceedings of the 1994 IEEE International Conference on Robotics and Automation.

[37]  Surya P. N. Singh,et al.  V-REP: A versatile and scalable robot simulation framework , 2013, 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[38]  S. LaValle.  Rapidly-exploring random trees : a new tool for path planning , 1998 .

[39]  Mark H. Overmars,et al.  A random approach to motion planning , 1992 .

[40]  and as an in , 2022 .