Factory: Fast Contact for Robotic Assembly

—Robotic assembly is one of the oldest and most chal- lenging applications of robotics. In other areas of robotics, such as perception and grasping, simulation has rapidly accelerated research progress, particularly when combined with modern deep learning. However, accurately, efficiently, and robustly simulating the range of contact-rich interactions in assembly remains a longstanding challenge. In this work, we present Factory , a set of physics simulation methods and robot learning tools for such applications. We achieve real-time or faster simulation of a wide range of contact-rich scenes, including simultaneous simulation of 1000 nut-and-bolt interactions. We provide 60 carefully- designed part models, 3 robotic assembly environments, and 7 robot controllers for training and testing virtual robots. Finally, we train and evaluate proof-of-concept reinforcement learning policies for nut-and-bolt assembly. We aim for Factory to open the doors to using simulation for robotic assembly, as well as many other contact-rich applications in robotics. Please see our website for supplementary content, including videos. 1

[1]  Yashraj S. Narang,et al.  DefGraspSim: Physics-Based Simulation of Grasp Outcomes for 3D Deformable Objects , 2022, IEEE Robotics and Automation Letters.

[2]  Masayoshi Tomizuka,et al.  Learning Insertion Primitives with Discrete-Continuous Hybrid Action Space for Robotic Assembly Tasks , 2021, 2022 International Conference on Robotics and Automation (ICRA).

[3]  Viktor Makoviychuk,et al.  OSCAR: Data-Driven Operational Space Control for Adaptive and Robust Robot Manipulation , 2021, 2022 International Conference on Robotics and Automation (ICRA).

[4]  Zilin Si,et al.  Taxim: An Example-based Simulation Model for GelSight Tactile Sensors , 2021, IEEE Robotics and Automation Letters.

[5]  R. Calandra,et al.  TACTO: A Fast, Flexible, and Open-Source Simulator for High-Resolution Vision-Based Tactile Sensors , 2020, IEEE Robotics and Automation Letters.

[6]  S. Schaal,et al.  Residual Learning from Demonstration: Adapting DMPs for Contact-rich Manipulation , 2020, IEEE Robotics and Automation Letters.

[7]  Lin Shao,et al.  RoboAssembly: Learning Generalizable Furniture Assembly Policy in a Novel Multi-robot Contact-rich Simulation Environment , 2021, ArXiv.

[8]  Maya Cakmak,et al.  Assistive Tele-op: Leveraging Transformers to Collect Robotic Task Demonstrations , 2021, ArXiv.

[9]  Joseph J. Lim,et al.  Adversarial Skill Chaining for Long-Horizon Robot Manipulation via Terminal State Regularization , 2021, CoRL.

[10]  Pulkit Agrawal,et al.  A System for General In-Hand Object Re-Orientation , 2021, CoRL.

[11]  Jing Xu,et al.  Data-Efficient Hierarchical Reinforcement Learning for Robotic Assembly Control Applications , 2021, IEEE Transactions on Industrial Electronics.

[12]  Philipp Reist,et al.  Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning , 2021, CoRL.

[13]  Miles Macklin,et al.  Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning , 2021, NeurIPS Datasets and Benchmarks.

[14]  Manuel Wüthrich,et al.  Transferring Dexterous Manipulation from GPU Simulation to a Remote Real-World TriFinger , 2021, ArXiv.

[15]  Kenny Erleben,et al.  Contact and friction simulation for computer graphics , 2021, SIGGRAPH Courses.

[16]  Silvio Savarese,et al.  iGibson 2.0: Object-Centric Simulation for Robot Learning of Everyday Household Tasks , 2021, CoRL.

[17]  Timothy R. Langlois,et al.  Intersection-free rigid body dynamics , 2021, ACM Transactions on Graphics.

[18]  Jens Lambrecht,et al.  Towards Real-World Force-Sensitive Robotic Assembly through Deep Reinforcement Learning in Simulations , 2021, 2021 IEEE/ASME International Conference on Advanced Intelligent Mechatronics (AIM).

[19]  Yashraj S. Narang,et al.  DefGraspSim: Simulation-based grasping of 3D deformable objects , 2021, ArXiv.

[20]  Angel X. Chang,et al.  Habitat 2.0: Training Home Assistants to Rearrange their Habitat , 2021, NeurIPS.

[21]  Masayoshi Tomizuka,et al.  Trajectory Optimization for Manipulation of Deformable Objects: Assembly of Belt Drive Units , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[22]  Shuran Song,et al.  FlingBot: The Unreasonable Effectiveness of Dynamic Manipulation for Cloth Unfolding , 2021, ArXiv.

[23]  Fabio Gramazio,et al.  Robotic assembly of timber joints using reinforcement learning , 2021, Automation in Construction.

[24]  Roozbeh Mottaghi,et al.  ManipulaTHOR: A Framework for Visual Object Manipulation , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[25]  Yashraj S. Narang,et al.  Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projections , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[26]  Oleg O. Sushkov,et al.  Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study , 2021, Robotics: Science and Systems.

[27]  Stefan Schaal,et al.  Benchmarking Off-The-Shelf Solutions to Robotic Assembly Tasks , 2021, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[28]  Michelle A. Lee,et al.  Interpreting Contact Interactions to Overcome Failure in Robot Assembly Tasks , 2021, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[29]  Dieter Fox,et al.  ACRONYM: A Large-Scale Grasp Dataset Based on Simulation , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[30]  Masayoshi Tomizuka,et al.  Learning Dense Rewards for Contact-Rich Manipulation Tasks , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[31]  Quang-Cuong Pham,et al.  Learning Sequences of Manipulation Primitives for Robotic Assembly , 2020, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[32]  Jieliang Luo,et al.  A Learning Approach to Robot-Agnostic Force-Guided High Precision Assembly , 2020, 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[33]  Danica Kragic,et al.  Stability-Guaranteed Reinforcement Learning for Contact-Rich Manipulation , 2020, IEEE Robotics and Automation Letters.

[34]  Joseph J. Lim,et al.  IKEA Furniture Assembly Environment for Long-Horizon Complex Manipulation Tasks , 2019, 2021 IEEE International Conference on Robotics and Automation (ICRA).

[35]  Christian Duriez,et al.  On the use of simulation in robotics: Opportunities, challenges, and suggestions for moving forward , 2020, Proceedings of the National Academy of Sciences.

[36]  Maria Bauza,et al.  Tactile Object Pose Estimation from the First Touch with Geometric Contact Rendering , 2020, CoRL.

[37]  Yashraj S. Narang,et al.  STReSSD: Sim-To-Real from Sound for Stochastic Dynamics , 2020, CoRL.

[38]  Oliver Kroemer,et al.  Towards Robotic Assembly by Predicting Robust, Precise and Task-oriented Grasps , 2020, CoRL.

[39]  Dongjun Lee,et al.  Sim-to-Real Transfer of Bolting Tasks with Tight Tolerance , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[40]  Lorenz Wellhausen,et al.  Learning quadrupedal locomotion over challenging terrain , 2020, Science Robotics.

[41]  Byron Boots,et al.  Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion , 2020, CoRL.

[42]  Miriam Zacksenhouse,et al.  Deep Reinforcement Learning for Contact-Rich Skills Using Compliant Movement Primitives , 2020, ArXiv.

[43]  K. Harada,et al.  Variable Compliance Control for Robotic Peg-in-Hole Assembly: A Deep Reinforcement Learning Approach , 2020, Applied Sciences.

[44]  Mathieu Aubry,et al.  CosyPose: Consistent multi-view multi-object 6D pose estimation , 2020, ECCV.

[45]  Sergey Levine,et al.  Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[46]  ErlebenKenny,et al.  Local Optimization for Robust Signed Distance Field Collision , 2020, Proc. ACM Comput. Graph. Interact. Tech..

[47]  Afsoon Afzal,et al.  A Study on the Challenges of Using Robotics Simulators for Testing , 2020, ArXiv.

[48]  Yu Sun,et al.  Benchmarking Protocols for Evaluating Small Parts Robotic Assembly Systems , 2020, IEEE Robotics and Automation Letters.

[49]  Leonidas J. Guibas,et al.  SAPIEN: A SimulAted Part-Based Interactive ENvironment , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[50]  Yashraj S. Narang,et al.  Inferring the Material Properties of Granular Media for Robotic Tasks , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[51]  Nikolaus Correll,et al.  Robots assembling machines: learning from the World Robot Summit 2018 Assembly Challenge , 2020, Adv. Robotics.

[52]  Jeannette Bohg,et al.  Learning to Scaffold the Development of Robotic Manipulation Skills , 2019, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[53]  Silvio Savarese,et al.  Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks , 2019, IEEE Transactions on Robotics.

[54]  Jakub W. Pachocki,et al.  Learning dexterous in-hand manipulation , 2018, Int. J. Robotics Res..

[55]  Mabel M. Zhang Necessity for More Realistic Contact Simulation , 2020 .

[56]  Timothy R. Langlois,et al.  Incremental Potential Contact: Intersection- and Inversion-free, Large-Deformation Dynamics , 2020 .

[57]  Hui Li,et al.  Dynamic Experience Replay , 2019, CoRL.

[58]  P. Allen,et al.  MAT: Multi-Fingered Adaptive Tactile Grasping via Deep Reinforcement Learning , 2019, CoRL.

[59]  Stefan Jeschke,et al.  Small steps in physics simulation , 2019, Symposium on Computer Animation.

[60]  Silvio Savarese,et al.  Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks , 2019, 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[61]  Myungsin Kim,et al.  Data-Driven Contact Clustering for Robot Simulation , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[62]  Alice M. Agogino,et al.  Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[63]  Jan Bender,et al.  Interlinked SPH Pressure Solvers for Strong Fluid-Rigid Coupling , 2019, ACM Trans. Graph..

[64]  Ken Goldberg,et al.  Learning ambidextrous robot grasping policies , 2019, Science Robotics.

[65]  Peter C. Horak,et al.  On the Similarities and Differences Among Contact Models in Robot Simulation , 2019, IEEE Robotics and Automation Letters.

[66]  Matthew T. Mason,et al.  A Survey of Automated Threaded Fastening , 2019, IEEE Transactions on Automation Science and Engineering.

[67]  Yevgen Chebotar,et al.  Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[68]  Oleg O. Sushkov,et al.  A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learning , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[69]  Masayoshi Tomizuka,et al.  A Learning Framework for High Precision Industrial Assembly , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[70]  Yu Sun,et al.  A dataset of daily interactive manipulation , 2018, Int. J. Robotics Res..

[71]  Silvio Savarese,et al.  ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation , 2018, CoRL.

[72]  N. Hogan,et al.  Impedance and Interaction Control , 2018 .

[73]  Alice M. Agogino,et al.  Deep Reinforcement Learning for Robotic Assembly of Mixed Deformable and Rigid Objects , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[74]  Xian Zhou,et al.  Can robots assemble an IKEA chair? , 2018, Science Robotics.

[75]  Pieter Abbeel,et al.  Learning Robotic Assembly from CAD , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[76]  Giovanni De Magistris,et al.  Deep reinforcement learning for high precision assembly tasks , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[77]  Martin A. Riedmiller,et al.  Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards , 2017, ArXiv.

[78]  Glen Berseth,et al.  DeepLoco , 2017, ACM Trans. Graph..

[79]  Wojciech Zaremba,et al.  Domain randomization for transferring deep neural networks from simulation to the real world , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[80]  Michiel van de Panne,et al.  Learning locomotion skills using DeepRL: does the choice of action space matter? , 2016, Symposium on Computer Animation.

[81]  Kevin M. Lynch,et al.  Modern Robotics: Mechanics, Planning, and Control , 2017 .

[82]  Alessio Rocchi,et al.  Stable simulation of underactuated compliant hands , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[83]  Eric Lengyel Volumetric Hierarchical Approximate Convex Decomposition , 2016 .

[84]  Hammad Mazhar,et al.  Chrono: An Open Source Multi-physics Dynamics Engine , 2015, HPCSE.

[85]  Hongyi Xu,et al.  Implicit Multibody Penalty-BasedDistributed Contact , 2014, IEEE Transactions on Visualization and Computer Graphics.

[86]  Blaine Lilly,et al.  Mechanical Assemblies: their Design, Manufacture, and Role in Product Development , 2013 .

[87]  Kris Hauser,et al.  Robust Contact Generation for Robot Simulation with Unstructured Meshes , 2013, ISRR.

[88]  Maud Marchal,et al.  Efficient collision detection for brittle fracture , 2012, SCA '12.

[89]  Jeong Kim,et al.  Finite element analysis and modeling of structure with bolted joints , 2007 .

[90]  C. Lacoursière Ghosts and machines : regularized variational methods for interactive simulations of multibodies with dry frictional contacts , 2007 .

[91]  Ming C. Lin,et al.  A modular haptic rendering algorithm for stable and transparent 6-DOF manipulation , 2006, IEEE Transactions on Robotics.

[92]  S. Buss Introduction to Inverse Kinematics with Jacobian Transpose , Pseudoinverse and Damped Least Squares methods , 2004 .

[93]  Matthew T. Mason,et al.  Mechanics of Robotic Manipulation , 2001 .

[94]  Louis A. Martin-Vega,et al.  Industrial perspective on research needs and opportunities in manufacturing assembly , 1995 .

[95]  Oussama Khatib,et al.  A unified approach for motion and force control of robot manipulators: The operational space formulation , 1987, IEEE J. Robotics Autom..