Testing Neural Network Classifiers Based on Metamorphic Relations

The application of machine learning programs is becoming increasingly more widespread. Neural networks, which are among the most popular machine learning programs, play important roles in people's daily lives, such as by controlling cars in autonomous driving systems. However, neural networks still lack effective testing methods. To address this problem, this paper proposes a testing method for neural network classifiers based on metamorphic relations. Firstly, it designs metamorphic relations to transform the original data set into derivative data sets. Then, it uses the data before and after the transformation to train and test the neural network classifier, respectively. Finally, it checks whether the output conforms to the metamorphic relations. The neural network classifier is defective if conflicts are detected. Experiments are conducted on a neural network classifier from Stanford's cs231n course to verify the effectiveness of the method. The results show that the defect detection capability of the proposed method is accurate, and 87.5% of the mutants are successfully detected.