Safe Driving via Expert Guided Policy Optimization

When learning common skills like driving, beginners usually have 1 experienced people or domain experts standing by to ensure the safety of the 2 learning process. We formulate such learning scheme under the Expert-in-the-loop 3 Reinforcement Learning (ERL) where a guardian is introduced to safeguard the 4 exploration of the learning agent. While allowing the sufficient exploration in the 5 uncertain environment, the guardian will intervene under dangerous situations and 6 demonstrate the correct actions to avoid the potential accident. Thus ERL enables 7 both exploration and expert’s partial demonstration as two training data sources. 8 Following such new setting, we develop a novel Expert Guided Policy Optimization 9 (EGPO) method. This method integrates the guardian in the loop of reinforcement 10 learning, which is composed of an expert policy to generate demonstration and a 11 switch function to decide when to intervene. Particularly, constrained optimization 12 technique is used to tackle the trivial solution that the agent deliberately behaves 13 dangerously to deceive the expert into taking over all the time. Offline RL technique 14 is further used to learn from the partial demonstrations generated by the expert. 15 Safe driving experiments show that our method achieves superior training and 16 test-time safety, outperforms baselines with a large margin in sample efficiency, 17 and preserves the generalization capacity to unseen environments in test-time 1. 18