Design and Development of an automated Robotic Pick & Stow System for an e-Commerce Warehouse

In this paper, we provide details of a robotic system that can automate the task of picking and stowing objects from and to a rack in an e-commerce fulfillment warehouse. The system primarily comprises of four main modules: (1) Perception module responsible for recognizing query objects and localizing them in the 3-dimensional robot workspace; (2) Planning module generates necessary paths that the robot end- effector has to take for reaching the objects in the rack or in the tote; (3) Calibration module that defines the physical workspace for the robot visible through the on-board vision system; and (4) Gripping and suction system for picking and stowing different kinds of objects. The perception module uses a faster region-based Convolutional Neural Network (R-CNN) to recognize objects. We designed a novel two finger gripper that incorporates pneumatic valve based suction effect to enhance its ability to pick different kinds of objects. The system was developed by IITK-TCS team for participation in the Amazon Picking Challenge 2016 event. The team secured a fifth place in the stowing task in the event. The purpose of this article is to share our experiences with students and practicing engineers and enable them to build similar systems. The overall efficacy of the system is demonstrated through several simulation as well as real-world experiments with actual robots.

[1]  Charles L. Lawson,et al.  Solving least squares problems , 1976, Classics in applied mathematics.

[2]  Joseph Redmon,et al.  Real-time grasp detection using convolutional neural networks , 2014, 2015 IEEE International Conference on Robotics and Automation (ICRA).

[3]  G. Oriolo,et al.  Robotics: Modelling, Planning and Control , 2008 .

[4]  Aude Billard,et al.  Active Teaching in Robot Programming by Demonstration , 2007, RO-MAN 2007 - The 16th IEEE International Symposium on Robot and Human Interactive Communication.

[5]  Kenneth Y. Goldberg,et al.  Cloud-based robot grasping with the google object recognition engine , 2013, 2013 IEEE International Conference on Robotics and Automation.

[6]  Wen Ding,et al.  Study of Smart Warehouse Management System Based on the IOT , 2013 .

[7]  Peter I. Corke,et al.  Towards Vision-Based Deep Reinforcement Learning for Robotic Motion Control , 2015, ICRA 2015.

[8]  Ashutosh Saxena,et al.  Robotic Grasping of Novel Objects using Vision , 2008, Int. J. Robotics Res..

[9]  Yuting Zhang,et al.  Improving object detection with deep convolutional networks via Bayesian optimization and structured prediction , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[10]  Yuval Tassa,et al.  Continuous control with deep reinforcement learning , 2015, ICLR.

[11]  Tamio Arai,et al.  Teaching of grasp/graspless manipulation for industrial robots by human demonstration , 2002, IEEE/RSJ International Conference on Intelligent Robots and Systems.

[12]  Lydia E. Kavraki,et al.  The Open Motion Planning Library , 2012, IEEE Robotics & Automation Magazine.

[13]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[14]  Oliver Otto,et al.  Constructing a Gazebo: Supporting Teamwork in a Tightly Coupled, Distributed Task in Virtual Reality , 2003, Presence: Teleoperators & Virtual Environments.

[15]  Karmele López de Ipiña,et al.  State Machine Based Architecture to Increase Flexibility of Dual-Arm Robot Programming , 2015, IWINAC.

[16]  Mark W. Spong,et al.  Robot dynamics and control , 1989 .

[17]  Steven M. LaValle,et al.  Rapidly-Exploring Random Trees: Progress and Prospects , 2000 .

[18]  Kostas E. Bekris,et al.  A Dataset for Improved RGBD-Based Object Detection and Pose Estimation for Warehouse Pick-and-Place , 2015, IEEE Robotics and Automation Letters.

[19]  Reinhard Klein,et al.  Efficient RANSAC for Point‐Cloud Shape Detection , 2007, Comput. Graph. Forum.

[20]  T. Rabbani,et al.  SEGMENTATION OF POINT CLOUDS USING SMOOTHNESS CONSTRAINT , 2006 .

[21]  Bill Triggs,et al.  Histograms of oriented gradients for human detection , 2005, 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05).

[22]  Roger Y. Tsai,et al.  A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses , 1987, IEEE J. Robotics Autom..

[23]  幸康 堂前.  Amazon Picking Challenge 2016 , 2016 .

[24]  Jiri Matas,et al.  Robust Detection of Lines Using the Progressive Probabilistic Hough Transform , 2000, Comput. Vis. Image Underst..

[25]  Michael Riis Andersen,et al.  Kinect Depth Sensor Evaluation for Computer Vision Applications , 2012 .

[26]  B. Faverjon,et al.  Probabilistic Roadmaps for Path Planning in High-Dimensional Con(cid:12)guration Spaces , 1996 .

[27]  Raffaello D'Andrea,et al.  Coordinating Hundreds of Cooperative, Autonomous Vehicles in Warehouses , 2007, AI Mag..

[28]  Ming Yang,et al.  Magnetic Sensing System Design for Intelligent Vehicle Guidance , 2010, IEEE/ASME Transactions on Mechatronics.

[29]  Angappa Gunasekaran,et al.  Bottom-Up Approach based on Internet of things for Order Fulfillment in a Collaborative Warehousing Environment , 2015 .

[30]  Oliver Brock,et al.  Lessons from the Amazon Picking Challenge: Four Aspects of Building Robotic Systems , 2016, IJCAI.

[31]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[32]  Z. Zivkovic.  Improved adaptive Gaussian mixture model for background subtraction , 2004, ICPR 2004.

[33]  G. P. Otto,et al.  "Region-growing" algorithm for matching of terrain images , 1989, Image Vis. Comput..

[34]  Yong-Tae Kim,et al.  Navigation Method of the Transportation Robot Using Fuzzy Line Tracking and QR Code Recognition , 2017, Int. J. Humanoid Robotics.

[35]  Kate Saenko,et al.  High precision grasp pose detection in dense clutter , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[36]  Wolfram Burgard,et al.  OctoMap : A Probabilistic , Flexible , and Compact 3 D Map Representation for Robotic Systems , 2010 .

[37]  Steven Lake Waslander,et al.  Planning Paths for Package Delivery in Heterogeneous Multirobot Teams , 2015, IEEE Transactions on Automation Science and Engineering.

[38]  Jonathan Bohren,et al.  The SMACH High-Level Executive [ROS News] , 2010 .

[39]  K. S. Arun,et al.  Least-Squares Fitting of Two 3-D Point Sets , 1987, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[40]  Zhengyou Zhang,et al.  Microsoft Kinect Sensor and Its Effect , 2012, IEEE Multim..

[41]  Raffaello D'Andrea,et al.  Rapyuta: A Cloud Robotics Platform , 2015, IEEE Transactions on Automation Science and Engineering.

[42]  Jitendra Malik,et al.  Region-Based Convolutional Networks for Accurate Object Detection and Segmentation , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[43]  Joseph M. Romano,et al.  The Amazon Picking Challenge , 2016, AI Mag..

[44]  Jean-Daniel Boissonnat,et al.  A practical exact motion planning algorithm for polygonal objects amidst polygonal obstacles , 1988, Proceedings. 1988 IEEE International Conference on Robotics and Automation.

[45]  Martijn Wisse,et al.  Team Delft's Robot Winner of the Amazon Picking Challenge 2016 , 2016, RoboCup.

[46]  Jitendra Malik,et al.  Shape matching and object recognition using shape contexts , 2010, 2010 3rd International Conference on Computer Science and Information Technology.

[47]  Niloy J. Mitra,et al.  Estimating surface normals in noisy point cloud data , 2003, SCG '03.

[48]  Ronald M. Summers,et al.  Deep Convolutional Neural Networks for Computer-Aided Detection: CNN Architectures, Dataset Characteristics and Transfer Learning , 2016, IEEE Transactions on Medical Imaging.

[49]  Yuxin Peng,et al.  Error-Driven Incremental Learning in Deep Convolutional Neural Network for Large-Scale Image Classification , 2014, ACM Multimedia.

[50]  Daniel E. Koditschek,et al.  Robot planning and control via potential functions , 1989 .

[51]  MengChu Zhou,et al.  An integrated e-supply chain model for agile and environmentally conscious manufacturing , 2001 .

[52]  Francis L. Merat,et al.  Introduction to robotics: Mechanics and control , 1987, IEEE J. Robotics Autom..

[53]  K. Ohnishi,et al.  Eye-to-hand approach on eye-in-hand configuration within real-time visual servoing , 2004, IEEE/ASME Transactions on Mechatronics.

[54]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[55]  Zhengyou Zhang,et al.  A Flexible New Technique for Camera Calibration , 2000, IEEE Trans. Pattern Anal. Mach. Intell..

[56]  Anil K. Jain,et al.  Object detection using gabor filters , 1997, Pattern Recognit..

[57]  Jean-Claude Latombe,et al.  Numerical potential field techniques for robot path planning , 1991, Fifth International Conference on Advanced Robotics 'Robots in Unstructured Environments.

[58]  Jan Rosell,et al.  The Kautham project: A teaching and research tool for robot motion planning , 2014, Proceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA).

[59]  Dinesh Manocha,et al.  FCL: A general purpose library for collision and proximity queries , 2012, 2012 IEEE International Conference on Robotics and Automation.

[60]  Lorenzo Sabattini,et al.  Ensemble Coordination Approach in Multi-AGV Systems Applied to Industrial Warehouses , 2015, IEEE Transactions on Automation Science and Engineering.

[61]  Morgan Quigley,et al.  ROS: an open-source Robot Operating System , 2009, ICRA 2009.

[62]  Narendra Ahuja,et al.  A potential field approach to path planning , 1992, IEEE Trans. Robotics Autom..

[63]  Kuan-Ting Yu,et al.  A Summary of Team MIT's Approach to the Amazon Picking Challenge 2015 , 2016, ArXiv.

[64]  Lydia Tapia,et al.  Efficient Motion-based Task Learning for a Serial Link Manipulator , 2014 .

[65]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[66]  Robert Platt,et al.  Using Geometry to Detect Grasps in 3D Point Clouds , 2015, 1501.03100.

[67]  Peter Kovesi,et al.  Shapelets correlated with surface normals produce surfaces , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[68]  Honglak Lee,et al.  Deep learning for detecting robotic grasps , 2013, Int. J. Robotics Res..

[69]  James J. Kuffner,et al.  OpenRAVE: A Planning Architecture for Autonomous Robotics , 2008 .

[70]  Yangmin Li,et al.  Inverse Kinematics and Control of a 7-DOF Redundant Manipulator Based on the Closed-Loop Algorithm , 2010 .

[71]  Frank Lingelbach,et al.  Path planning using probabilistic cell decomposition , 2004, IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA '04. 2004.

[72]  Markus Vincze,et al.  Learning grasps for unknown objects in cluttered scenes , 2013, 2013 IEEE International Conference on Robotics and Automation.

[73]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[74]  Sergey Levine,et al.  Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection , 2016, Int. J. Robotics Res..

[75]  David A. McAllester,et al.  A discriminatively trained, multiscale, deformable part model , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[76]  G LoweDavid,et al.  Distinctive Image Features from Scale-Invariant Keypoints , 2004 .

[77]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.