OCRTOC: A Cloud-Based Competition and Benchmark for Robotic Grasping and Manipulation

In this paper, we propose a cloud-based benchmark for robotic grasping and manipulation, called the OCRTOC benchmark. The benchmark focuses on the object rearrangement problem, specifically table organization tasks. We provide a set of identical real robot setups and facilitate remote experiments of standardized table organization scenarios in varying difficulties. In this workflow, users upload their solutions to our remote server and their code is executed on the real robot setups and scored automatically. After each execution, the OCRTOC team resets the experimental setup manually. We also provide a simulation environment that researchers can use to develop and test their solutions. With the OCRTOC benchmark, we aim to lower the barrier of conducting reproducible research on robotic grasping and manipulation and accelerate progress in this field. Executing standardized scenarios on identical real robot setups allows us to quantify algorithm performances and achieve fair comparisons. Using this benchmark we held a competition in the 2020 International Conference on Intelligence Robots and Systems (IROS 2020). In total, 59 teams took part in this competition worldwide. We present the results and our observations of the 2020 competition, and discuss our adjustments and improvements for the upcoming OCRTOC 2021 competition. The homepage of the OCRTOC competition is www.ocrtoc.org, and the OCRTOC software package is available at https://github.com/OCRTOC/OCRTOC_software_package.

[1]  Xinyu Liu,et al.  Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics , 2017, Robotics: Science and Systems.

[2]  Roozbeh Mottaghi,et al.  Rearrangement: A Challenge for Embodied AI , 2020, ArXiv.

[3]  Danica Kragic,et al.  Benchmarking Bimanual Cloth Manipulation , 2020, IEEE Robotics and Automation Letters.

[4]  Andrew J. Davison,et al.  RLBench: The Robot Learning Benchmark & Learning Environment , 2019, IEEE Robotics and Automation Letters.

[5]  Abderrahmane Kheddar,et al.  Impact-Friendly Robust Control Design with Task-Space Quadratic Optimization , 2019, Robotics: Science and Systems.

[6]  Aude Billard,et al.  Benchmark for Bimanual Robotic Manipulation of Semi-Deformable Objects , 2020, IEEE Robotics and Automation Letters.

[7]  Vincent Lepetit,et al.  Model Based Training, Detection and Pose Estimation of Texture-Less 3D Objects in Heavily Cluttered Scenes , 2012, ACCV.

[8]  Kaiyu Hang,et al.  Benchmarking Cluttered Robot Pick-and-Place Manipulation With the Box and Blocks Test , 2020, IEEE Robotics and Automation Letters.

[9]  P. Abbeel,et al.  Yale-CMU-Berkeley dataset for robotic manipulation research , 2017, Int. J. Robotics Res..

[10]  Ziyuan Liu,et al.  Table-Top Scene Analysis Using Knowledge-Supervised MCMC , 2015, ArXiv.

[11]  Geng Gao,et al.  Modular, Accessible, Sensorized Objects for Evaluating the Grasping and Manipulation Capabilities of Grippers and Hands , 2020, IEEE Robotics and Automation Letters.

[12]  Leonidas J. Guibas,et al.  SAPIEN: A SimulAted Part-Based Interactive ENvironment , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Silvio Savarese,et al.  ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation , 2018, CoRL.

[14]  Ken Goldberg,et al.  GOMP: Grasp-Optimized Motion Planning for Bin Picking , 2020, 2020 IEEE International Conference on Robotics and Automation (ICRA).

[15]  Ugo Pattacini,et al.  GRASPA 1.0: GRASPA is a Robot Arm graSping Performance BenchmArk , 2020, IEEE Robotics and Automation Letters.

[16]  Yu Sun,et al.  Robotic Grasping and Manipulation Competition: Task Pool , 2016 .

[17]  Silvio Savarese,et al.  DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[18]  Kaiyu Hang,et al.  Benchmarking Protocol for Grasp Planning Algorithms , 2020, IEEE Robotics and Automation Letters.

[19]  Oliver Brock,et al.  Lessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems , 2016, IJCAI.

[20]  Leslie Pack Kaelbling,et al.  PDDLStream: Integrating Symbolic Planners and Blackbox Samplers via Optimistic Adaptive Planning , 2020, ICAPS.

[21]  Elena R. Messina,et al.  Performance Metrics and Test Methods for Robotic Hands (Draft) , 2018 .

[22]  Li Wang,et al.  The Robotarium: A remotely accessible swarm robotics research testbed , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[23]  Cewu Lu,et al.  GraspNet-1Billion: A Large-Scale Benchmark for General Object Grasping , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[24]  Rainer Bischoff,et al.  Robotics Education and Research at Scale: A Remotely Accessible Robotics Development Platform , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[25]  Sergey Levine,et al.  REPLAB: A Reproducible Low-Cost Arm Benchmark for Robotic Learning , 2019, 2019 International Conference on Robotics and Automation (ICRA).

[26]  Lydia E. Kavraki,et al.  Platform-Independent Benchmarks for Task and Motion Planning , 2018, IEEE Robotics and Automation Letters.

[27]  Henry Zhu,et al.  ROBEL: Robotics Benchmarks for Learning with Low-Cost Robots , 2019, CoRL.

[28]  Abderrahmane Kheddar,et al.  Quadratic Programming for Multirobot and Task-Space Force Control , 2019, IEEE Transactions on Robotics.

[29]  Dieter Fox,et al.  PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes , 2017, Robotics: Science and Systems.

[30]  Ning Xu,et al.  YouTube-VOS: Sequence-to-Sequence Video Object Segmentation , 2018, ECCV.

[31]  Oliver Brock,et al.  Lessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems , 2016, Robotics: Science and Systems.

[32]  Larry Jackel,et al.  The DARPA Robotics Challenge Finals: Results and Perspectives , 2017, J. Field Robotics.