Planning with Closed Loop Macro Actions

Planning with ClosedLo op Macro ActionsDoina PrecupUniversityof MassachusettsAmherst MA http wwwcsumassedu dprecupRichard S SuttonyofMassacttp wwwcsumassedu ricSatinder Singhy of ColoradoBoulder CO ttp wwwcscoloradoedu bavejaAbstractPlanning and learning at multiple levels of temp oral abstraction is a key problem for articial intelligence In this pap er we summarize an approachtothis problem based on the mathematical frameworkof Markov decision pro cesses and reinforcement learning Conventional mo delbased reinforcement learninguses primitive actions that last one time step and thatcan b e mo deled indep endently of the learning agentThese can b e generalized tomacro actionsmultistepactions sp ecied by an arbitrary p olicy and a wayofcompleting Macro actions generalize the classical notion of a macro op erator in that they are closed lo opuncertain and of variable duration Macro actions areneeded to represent commonsense higherlevel actionssuch as going to lunch grasping an ob ject or traveling to a distantcity This pap er generalizes prior workon temp orally abstract mo dels Sutton and extends it from the prediction setting to include actionscontrol and planning Wedene asemantics of mo dels of macro actions that guarantees the validityofplanning using such mo dels This pap er presentnewresults in the theory of planning with macro actionsand illustrates its p otential advantages in a gridworldtaskIntro ductionThe need for hierarchical and abstract planning is afundamental problem in AI see eg Sacerdoti Laird et al  Korf  Kaelbling  DayanHinton Mo delbased reinforcement learning oers a p ossible solution to the problem of integrating planning with realtime learning and decisionmaking Peng  Williams  Mo ore  Atkeson Sutton  Sutton  Barto  Howeverconventional mo delbased reinforcement learning usesonestep mo dels that cannot represent commonsensehigherlevel actionsMo deling such actions requiresthe ability to handle dierent interrelated levels oftemp oral abstractionSeveral researchers have prop osed extending reinforcement learning to a higher level by treating entireclosedlo op p olicies as actions whichwe callmacroactionseg Mahadevan  Connell  Singh Hub er  Grup en  Parr  Russell p ersonal communication Dietterich p ersonal communication McGovern Sutton  Fagg  Each macro action issp ecied by a closedlo op p olicywhich determines theprimitive actions when the macro action is in forceand by a completion function which determines whenthe macro action ends When the macro action completes a new primitive or macro action can b e selectedMacro actions are like AIs classical macro op erators in that they can takecontrol for some p erio dof time determining the actions during that time andin that one can cho ose among macro actions muchasone originally chose among primitive actions Howeverclassical macro op erators are only a xed sequenceof actions whereas macro actions incorp orate a general closedlo op p olicy and completion criterion Thesegeneralizations are required when the environmentissto chastic and uncertain with general goals as in reinforcement learning and Markov decision pro cessesThis pap er extends an approach to planning withmacro actions intro duced by Sutton  based onprior work by Singh  Dayan  and by Sutton and Pinette   This approach enables mo dels of the environment at dierent temp oral scales tobe intermixed pro ducing temp orally abstract mo delsSutton  Dayan  and Sutton and Pinette  were concerned only with predicting the environment in eect mo deling a single macro action LikeSingh  we mo del a whole set of macro actionsand consider choices among them but the extensionsummarized here includes more general macro actionsand control of the environment Wedevelop a generaltheory of mo deling and planning with macro actionsTo illustrate the kind of advance we are trying tomake consider the example task depicted in Figure