In autonomous vehicle (AV) control, allowing mistakes can be quite dangerous and costly in the real world. For this reason we investigate methods of training an AV without allowing the agent to explore and instead having a human explorer collect the data. Supervised learning has been explored for AV control, but it encounters the issue of the covariate shift. That is, training data collected from an optimal demonstration consists only of the states induced by the optimal control policy, but at runtime, the trained agent may encounter a vastly different state distribution with little relevant training data. To mitigate this issue, we have our human explorer make sub-optimal decisions. In order to have our agent not replicate these sub-optimal decisions, supervised learning requires that we either erase these actions, or replace these action with the correct action. Erasing is wasteful and replacing is difficult, since it is not easy to know the correct action without driving. We propose an alternate framework that includes continuous scalar feedback for each action, marking which actions we should replicate, which we should avoid, and how sure we are. Our framework learns continuous control from sub-optimal demonstration and evaluative feedback collected before training. We find that a human demonstrator can explore sub-optimal states in a safe manner, while still getting enough gradation to benefit learning. The collection method for data and feedback we call "Backseat Driver." We call the more general learning framework ReNeg, since it learns a regression from states to actions given negative as well as positive examples. We empirically validate several models in the ReNeg framework, testing on lane-following with limited data. We find that the best solution is a generalization of mean-squared error and outperforms supervised learning on the positive examples alone.
[1]
Wolfram Burgard,et al.
Learning driving styles for autonomous vehicles from demonstration
,
2015,
2015 IEEE International Conference on Robotics and Automation (ICRA).
[2]
Marc G. Bellemare,et al.
Safe and Efficient Off-Policy Reinforcement Learning
,
2016,
NIPS.
[3]
Guan Wang,et al.
Interactive Learning from Policy-Dependent Human Feedback
,
2017,
ICML.
[4]
Yang Gao,et al.
Reinforcement Learning from Imperfect Demonstrations
,
2018,
ICLR.
[5]
R. J. Williams,et al.
Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning
,
2004,
Machine Learning.
[6]
Xin Zhang,et al.
End to End Learning for Self-Driving Cars
,
2016,
ArXiv.
[7]
Pieter Abbeel,et al.
Apprenticeship learning via inverse reinforcement learning
,
2004,
ICML.
[8]
Carlo Tomasi,et al.
Distance Minimization for Reward Learning from Scored Trajectories
,
2016,
AAAI.
[9]
Guy Lever,et al.
Deterministic Policy Gradient Algorithms
,
2014,
ICML.
[10]
Martial Hebert,et al.
Learning monocular reactive UAV control in cluttered natural environments
,
2012,
2013 IEEE International Conference on Robotics and Automation.
[11]
J. Andrew Bagnell,et al.
Reinforcement and Imitation Learning via Interactive No-Regret Learning
,
2014,
ArXiv.
[12]
Byron Boots,et al.
Agile Off-Road Autonomous Driving Using End-to-End Deep Imitation Learning
,
2017,
ArXiv.
[13]
Martha White,et al.
Linear Off-Policy Actor-Critic
,
2012,
ICML.
[14]
Shimon Whiteson,et al.
Inverse Reinforcement Learning from Failure
,
2016,
AAMAS.
[15]
Byron Boots,et al.
Deeply AggreVaTeD: Differentiable Imitation Learning for Sequential Prediction
,
2017,
ICML.
[16]
Geoffrey J. Gordon,et al.
No-Regret Reductions for Imitation Learning and Structured Prediction
,
2010,
ArXiv.