Omnipush: accurate, diverse, real-world dataset of pushing dynamics with RGB-D video

Pushing is a fundamental robotic skill. Existing work has shown how to exploit models of pushing to achieve a variety of tasks, including grasping under uncertainty, in-hand manipulation and clearing clutter. Such models, however, are approximate, which limits their applicability.Learning-based methods can reason directly from raw sensory data with accuracy, and have the potential to generalize to a wider diversity of scenarios. However, developing and testing such methods requires rich-enough datasets. In this paper we introduce Omnipush, a dataset with high variety of planar pushing behavior.In particular, we provide 250 pushes for each of 250 objects, all recorded with RGB-D and a high precision tracking system. The objects are constructed so as to systematically explore key factors that affect pushing–the shape of the object and its mass distribution–which have not been broadly explored in previous datasets, and allow to study generalization in model learning.Omnipush includes a benchmark for meta-learning dynamic models, which requires algorithms that make good predictions and estimate their own uncertainty. We also provide an RGB video prediction benchmark and propose other relevant tasks that can be suited with this dataset. Data and code are available at https://web.mit.edu/mcube/omnipush-dataset/.

[1]  Sebastian Nowozin,et al.  Meta-Learning Probabilistic Inference for Prediction , 2018, ICLR.

[2]  Joshua B. Tenenbaum,et al.  Human-level concept learning through probabilistic program induction , 2015, Science.

[3]  Sergey Levine,et al.  Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables , 2019, ICML.

[4]  Mark R. Cutkosky,et al.  Practical Force-Motion Models for Sliding Manipulation , 1996, Int. J. Robotics Res..

[5]  Stefan Schaal,et al.  Combining learned and analytical models for predicting action effects from sensory data , 2017, Int. J. Robotics Res..

[6]  Soo-Hong Lee,et al.  Fixture planning with friction , 1991 .

[7]  T. Baumeister.  Combining Analytical and Learned Models for Model Predictive Control , 2018 .

[8]  Jitendra Malik,et al.  Learning to Poke by Poking: Experiential Learning of Intuitive Physics , 2016, NIPS.

[9]  Matthew T. Mason,et al.  Mechanics and Planning of Manipulator Pushing Operations , 1986 .

[10]  Sergey Levine,et al.  Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks , 2017, ICML.

[11]  Leslie Pack Kaelbling,et al.  Graph Element Networks: adaptive, structured computation and memory , 2019, ICML.

[12]  Sergey Levine,et al.  Probabilistic Model-Agnostic Meta-Learning , 2018, NeurIPS.

[13]  Katja Hofmann,et al.  Meta Reinforcement Learning with Latent Variable Gaussian Processes , 2018, UAI.

[14]  Alberto Rodriguez,et al.  GP-SUM. Gaussian Process Filtering of non-Gaussian Beliefs , 2018, WAFR.

[15]  Kazuo Tanie,et al.  Manipulation And Active Sensing By Pushing Using Tactile Feedback , 1992, Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems.

[16]  Kuan-Ting Yu,et al.  More than a million ways to be pushed. A high-fidelity experimental dataset of planar pushing , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[17]  Maria Bauzá,et al.  GP-SUM. Gaussian Processes Filtering of non-Gaussian Beliefs , 2017, ArXiv.

[18]  Sergey Levine,et al.  Stochastic Adversarial Video Prediction , 2018, ArXiv.

[19]  Maria Bauzá,et al.  A Data-Efficient Approach to Precise and Controlled Pushing , 2018, CoRL.

[20]  Amos J. Storkey,et al.  Towards a Neural Statistician , 2016, ICLR.

[21]  Leslie Pack Kaelbling,et al.  Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[22]  Yee Whye Teh,et al.  Attentive Neural Processes , 2019, ICLR.

[23]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[24]  Abdeslam Boularias,et al.  Information-theoretic Model Identification and Policy Search using Physics Engines with Application to Robotic Manipulation , 2017, ArXiv.

[25]  Alberto Rodriguez,et al.  Learning Synergies Between Pushing and Grasping with Self-Supervised Deep Reinforcement Learning , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[26]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[27]  Marcin Andrychowicz,et al.  Sim-to-Real Transfer of Robotic Control with Dynamics Randomization , 2017, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[28]  J. Andrew Bagnell,et al.  A convex polynomial model for planar sliding mechanics: theory, application, and experimental validation , 2018, Int. J. Robotics Res..

[29]  Dit-Yan Yeung,et al.  Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting , 2015, NIPS.

[30]  Yee Whye Teh,et al.  Meta reinforcement learning as task inference , 2019, ArXiv.

[31]  Oriol Vinyals,et al.  Matching Networks for One Shot Learning , 2016, NIPS.

[32]  Sergey Levine,et al.  Deep visual foresight for planning robot motion , 2016, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[33]  Eero P. Simoncelli,et al.  Image quality assessment: from error visibility to structural similarity , 2004, IEEE Transactions on Image Processing.

[34]  Giulio Sandini,et al.  A Vision-Based Learning Method for Pushing Manipulation , 1993 .

[35]  Rob Fergus,et al.  Stochastic Video Generation with a Learned Prior , 2018, ICML.

[36]  Manuela M. Veloso,et al.  Push-manipulation of complex passive mobile objects using experimentally acquired motion models , 2015, Auton. Robots.

[37]  A. Ruina,et al.  Planar sliding with dry friction Part 1. Limit surface and moment function , 1991 .

[38]  Yochai Blau,et al.  The Perception-Distortion Tradeoff , 2017, CVPR.

[39]  Pieter Abbeel,et al.  Policy transfer via modularity and reward guiding , 2017, 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[40]  Alexei A. Efros,et al.  The Unreasonable Effectiveness of Deep Features as a Perceptual Metric , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[41]  John Kenneth Salisbury,et al.  Pushing using learned manipulation maps , 2008, 2008 IEEE International Conference on Robotics and Automation.

[42]  Takeo Igarashi,et al.  Automatic learning of pushing strategy for delivery of irregular-shaped objects , 2011, 2011 IEEE International Conference on Robotics and Automation.

[43]  Yee Whye Teh,et al.  Conditional Neural Processes , 2018, ICML.

[44]  J. Andrew Bagnell,et al.  A convex polynomial force-motion model for planar sliding: Identification and application , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[45]  Alberto Rodriguez,et al.  Feedback Control of the Pusher-Slider System: A Story of Hybrid and Underactuated Contact Dynamics , 2016, WAFR.

[46]  Yoshua Bengio,et al.  Bayesian Model-Agnostic Meta-Learning , 2018, NeurIPS.

[47]  Sergey Levine,et al.  Learning to Adapt: Meta-Learning for Model-Based Control , 2018, ArXiv.

[48]  S. Srinivasa,et al.  Push-grasping with dexterous hands: Mechanics and a method , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[49]  Arthur C. Sanderson,et al.  The motion of a pushed, sliding workpiece , 1988, IEEE J. Robotics Autom..

[50]  Kevin M. Lynch,et al.  Stable Pushing: Mechanics, Controllability, and Planning , 1995, Int. J. Robotics Res..

[51]  Maria Bauza,et al.  A probabilistic data-driven model for planar pushing , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).