Visual closed-loop control for pouring liquids

Pouring a specific amount of liquid is a challenging task. In this paper we develop methods for robots to use visual feedback to perform closed-loop control for pouring liquids. We propose both a model-based and a model-free method utilizing deep learning for estimating the volume of liquid in a container. Our results show that the model-free method is better able to estimate the volume. We combine this with a simple PID controller to pour specific amounts of liquid, and show that the robot is able to achieve an average 38ml deviation from the target amount. To our knowledge, this is the first use of raw visual feedback to pour liquids in robotics.

[1]  S. Eddy.  Hidden Markov models. , 1996, Current opinion in structural biology.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Tsuhan Chen,et al.  Efficient feature extraction for 2D/3D objects in mesh representation , 2001, Proceedings 2001 International Conference on Image Processing (Cat. No.01CH37205).

[4]  Masayuki Inaba,et al.  Vision based behavior verification system of humanoid robot for daily environment tasks , 2006, 2006 6th IEEE-RAS International Conference on Humanoid Robots.

[5]  Larry Matthies,et al.  Daytime water detection based on color variation , 2010, 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[6]  Ales Ude,et al.  Learning to pour with a robot arm combining goal and shape learning for dynamic movement primitives , 2011, Robotics Auton. Syst..

[7]  Larry H. Matthies,et al.  Daytime water detection based on sky reflections , 2011, 2011 IEEE International Conference on Robotics and Automation.

[8]  Radu Bogdan Rusu,et al.  3D is here: Point Cloud Library (PCL) , 2011, 2011 IEEE International Conference on Robotics and Automation.

[9]  Trevor Darrell,et al.  A geometric approach to robotic laundry folding , 2012, Int. J. Robotics Res..

[10]  Yun Jiang,et al.  Learning to place new objects in a scene , 2012, Int. J. Robotics Res..

[11]  Maya Cakmak,et al.  Designing robot learners that ask good questions , 2012, 2012 7th ACM/IEEE International Conference on Human-Robot Interaction (HRI).

[12]  Toby P. Breckon,et al.  On Cross-Spectral Stereo Matching using Dense Gradient Features , 2012, BMVC.

[13]  Alexander Stoytchev,et al.  Object Categorization in the Sink : Learning Behavior – Grounded Object Categories with Water , 2012 .

[14]  Carme Torras,et al.  Force-based robot learning of pouring skills using parametric hidden Markov models , 2013, 9th International Workshop on Robot Motion and Control.

[15]  B. Wansink,et al.  Half Full or Empty: Cues That Lead Wine Drinkers to Unintentionally Overpour , 2014, Substance use & misuse.

[16]  Maya Cakmak,et al.  Enhanced robotic cleaning with a low-cost tool attachment , 2014, 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems.

[17]  Trevor Darrell,et al.  Caffe: Convolutional Architecture for Fast Feature Embedding , 2014, ACM Multimedia.

[18]  Krishnanand N. Kaipa,et al.  Incorporating Failure-to-Success Transitions in Imitation Learning for a Dynamic Pouring Task , 2014 .

[19]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[20]  Christopher G. Atkeson,et al.  Differential dynamic programming with temporally decomposed dynamics , 2015, 2015 IEEE-RAS 15th International Conference on Humanoid Robots (Humanoids).

[21]  Christopher G. Atkeson,et al.  Stereo vision of liquid and particle flow for robot pouring , 2016, 2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids).

[22]  Connor Schenck,et al.  Towards Learning to Perceive and Reason About Liquids , 2016, ISER.

[23]  Connor Schenck,et al.  Guided Policy Search with Delayed Senor Measurements , 2016, ArXiv.

[24]  Danfei Xu,et al.  Multi-sensor surface analysis for robotic ironing , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[25]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[26]  Michael Beetz,et al.  Envisioning the qualitative effects of robot manipulation actions using simulation-based projections , 2017, Artif. Intell..

[27]  Jürgen Schmidhuber,et al.  LSTM: A Search Space Odyssey , 2015, IEEE Transactions on Neural Networks and Learning Systems.