In this current research, Twin-Delayed DDPG (TD3) algorithm has been used to solve the most challenging virtual Artificial Intelligence application by training a 4-ant-legged robot as an Intelligent Agent to run across a field. Twin-Delayed DDPG (TD3) is an incredibly smart AI model of a Deep Reinforcement Learning which combines the state-of-the-art methods in Artificial Intelligence. These includes Policy gradient, Actor-Critics, and continuous Double Deep Q-Learning. These Deep Reinforcement Learning approaches trained an Intelligent agent to interact with an environment with automatic feature engineering, that is, necessitating minimal domain knowledge. For the implementation of the TD3, we used a two-layer feedforward neural network of 400 and 300 hidden nodes respectively, with Rectified Linear Units (ReLU) as an activation function between each layer for both the Actor and Critics. We, then added a final tanh unit after the output of the Actor. The Critic receives both the state and action as input to the first layer. Both the network parameters were updated using Adam optimizer. The idea behind the Twin-Delayed DDPG (TD3) is to reduce overestimation bias in Deep Q-Learning with discrete actions which are ineffective in an Actor-Critic domain setting. Based on the Maximum Average Reward over the evaluation time-step, our model achieved an approximate maximum of 2364. Therefore, we can truly say that, TD3 has obviously improved on both the learning speed and performance of the Deep Deterministic Policy Gradient (DDPG) in a challenging environment in a continuous control domain.
[1]
Yuval Tassa,et al.
Continuous control with deep reinforcement learning
,
2015,
ICLR.
[2]
Alex Graves,et al.
Playing Atari with Deep Reinforcement Learning
,
2013,
ArXiv.
[3]
Alec Radford,et al.
Proximal Policy Optimization Algorithms
,
2017,
ArXiv.
[4]
Sergey Levine,et al.
Trust Region Policy Optimization
,
2015,
ICML.
[5]
Yishay Mansour,et al.
Policy Gradient Methods for Reinforcement Learning with Function Approximation
,
1999,
NIPS.
[6]
Herke van Hoof,et al.
Addressing Function Approximation Error in Actor-Critic Methods
,
2018,
ICML.
[7]
Elman Mansimov,et al.
Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
,
2017,
NIPS.
[8]
Guy Lever,et al.
Deterministic Policy Gradient Algorithms
,
2014,
ICML.
[9]
David Silver,et al.
Deep Reinforcement Learning with Double Q-Learning
,
2015,
AAAI.
[10]
Jimmy Ba,et al.
Adam: A Method for Stochastic Optimization
,
2014,
ICLR.
[11]
Sergey Levine,et al.
Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
,
2018,
ICML.
[12]
Richard S. Sutton,et al.
Reinforcement Learning: An Introduction
,
1998,
IEEE Trans. Neural Networks.