Applying the Episodic Natural Actor-Critic Architecture to Motor Primitive Learning

, Stefan SchaalUniversity of Southern California, Los Angeles CA 90089, USAAbstract. Inthispaper, weinvestigatemotorprimitivelearning withtheNatural Actor-Critic approach. The Natural Actor-Critic consists out ofactor updates which are achieved using natural stochastic policy gradientswhile the critic obtains the natural policy gradient by linear regression.We show that this architecture can be used to learn the “building blocksof movement generation”, called motor primitives. Motor primitives areparameterizedcontrolpoliciessuchassplinesornonlineardifferentialequa-tions with desired attractor properties. We show that our most modernalgorithm, the Episodic Natural Actor-Critic outperforms previous algo-rithms by at least an order of magnitude. We demonstrate the efficiencyof this reinforcement learning method in the application of learning to hita baseball with an anthropomorphic robot arm.