Metamorphic Fuzz Testing of Autonomous Vehicles

Driving simulation is the primary approach for testing the software components of autonomous vehicles. This paper presents an automated testing method, termed metamorphic fuzz testing (MFT), in the context of simulation testing of autonomous driving. MFT differs from existing fuzzing techniques in the following two stages: First, it can generate "unrealistic" scenarios where scenes of the virtual world are refreshed frequently (so obstacles can suddenly appear / disappear)---this is to test the self-driving vehicle's robustness in the face of unexpected situations. In the second stage, MFT uses metamorphic relations as a filtering or debugging tool to distinguish between genuine failures and false alarms yielded in the first stage. We conduct empirical studies using the real-life Baidu Apollo self-driving system, recording a genuine failure rate of 3.7%. We have reported some of the detected failures to the Apollo team and received their confirmation. Our testing method is platform-independent and, therefore, can be applied to other autonomous driving systems and advanced driver-assistance systems (ADAS).