Practical Resolution Methods for MDPs in Robotics Exemplified With Disassembly Planning

In this letter, we focus on finding practical resolution methods for Markov decision processes (MDPs) in robotics. Some of the main difficulties of applying MDPs to real-world robotics problems are: first, having to deal with huge state spaces; and second, designing a method that is robust enough to dead ends. These complications restrict or make more difficult the application of methods, such as value iteration, policy iteration, or labeled real-time dynamic programming (LRTDP). We see in determinization and heuristic search a way to successfully work around these problems. In addition, we believe that many practical use cases offer the opportunity to identify hierarchies of subtasks and solve smaller, simplified problems. We propose a decision-making unit that operates in a probabilistic planning setting through stochastic shortest path problems, which generalize the most common types of MDPs. Our decision-making unit combines: first, automatic hierarchical organization of subtasks; and second, on-line resolution via determinization. We argue that several applications of planning benefit from these two strategies. We exemplify our approach with a robotized disassembly application. The disassembly problem is modeled in probabilistic planning definition language, and serves to define our experiments. Our results show many advantages of our method over LRTDP, such as a better capability to handle problems with large state spaces and state definitions that change when new fluents are discovered.

[1]  Bernd Scholz-Reiter,et al.  Flexible robot-based disassembly cell for obsolete TV-sets and monitors , 1999 .

[2]  Peng Dai,et al.  Reverse Iterative Deepening for Finite-Horizon MDPs with Large Branching Factors , 2012, ICAPS.

[3]  Rafael Aracil,et al.  Disassembly Planning Based on Precedence Relations among Assemblies , 2003 .

[4]  Alan Fern,et al.  Hindsight Optimization for Probabilistic Planning with Factored Actions , 2015, ICAPS.

[5]  Carme Torras,et al.  Interleaving Hierarchical Task Planning and Motion Constraint Testing for Dual-Arm Manipulation , 2018, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[6]  Robert Givan,et al.  FF-Replan: A Baseline for Probabilistic Planning , 2007, ICAPS.

[7]  Wheeler Ruml,et al.  Improving Determinization in Hindsight for On-line Probabilistic Planning , 2010, ICAPS.

[8]  Dinesh Manocha,et al.  D-Plan: Efficient Collision-Free Path Computation for Part Removal and Disassembly , 2008 .

[9]  Thomas Keller,et al.  PROST: Probabilistic Planning Based on UCT , 2012, ICAPS.

[10]  Mausam,et al.  Towards Scalable MDP Algorithms , 2011, IJCAI.

[11]  F. Teichteil-Königsbuch,et al.  RFF : A Robust , FF-Based MDP Planning Algorithm for Generating Policies with Low Probability of Failure , 2008 .

[12]  Sylvie Thiébaux,et al.  Probabilistic planning vs replanning , 2007 .

[13]  Subbarao Kambhampati,et al.  Probabilistic Planning via Determinization in Hindsight , 2008, AAAI.

[14]  Sami Kara,et al.  Robot Assisted Disassembly for the Recycling of Electric Vehicle Batteries , 2015 .

[15]  Håkan L. S. Younes,et al.  PPDDL 1 . 0 : An Extension to PDDL for Expressing Planning Domains with Probabilistic Effects , 2004 .

[16]  Surendra M. Gupta,et al.  Disassembly Sequencing Using Tabu Search , 2016, J. Intell. Robotic Syst..

[17]  Mausam,et al.  LRTDP Versus UCT for Online Probabilistic Planning , 2012, AAAI.

[18]  Mohamad Bdiwi,et al.  Autonomous disassembly of electric vehicle motors based on robot cognition , 2016, 2016 IEEE International Conference on Robotics and Automation (ICRA).

[19]  Leslie Pack Kaelbling,et al.  Integrated task and motion planning in belief space , 2013, Int. J. Robotics Res..

[20]  Craig A. Knoblock,et al.  PDDL-the planning domain definition language , 1998 .

[21]  Blai Bonet,et al.  mGPT: A Probabilistic Planner Based on Heuristic Search , 2005, J. Artif. Intell. Res..

[22]  Duan Guanghong,et al.  A Genetic Algorithm for Product Disassembly Sequence Planning , 2006, 2006 IEEE International Conference on Engineering of Intelligent Systems.

[23]  Malte Helmert,et al.  The Fast Downward Planning System , 2006, J. Artif. Intell. Res..

[24]  Rafael Aracil,et al.  Non-Destructive Disassembly Robot Cell for Demanufacturing Automation , 2003 .

[25]  Scott Sanner,et al.  Hindsight Optimization for Hybrid State and Action MDPs , 2017, AAAI.