Novelty-based Generalization Evaluation for Traffic Light Detection

The advent of Convolutional Neural Networks (CNNs) has led to their application in several domains. One noteworthy application is the perception system for autonomous driving that relies on the predictions from CNNs. Practitioners evaluate the generalization ability of such CNNs by calculating various metrics on an independent test dataset. A test dataset is often chosen based on only one precondition, i.e., its elements are not a part of the training data. But it may contain objects that are both similar and novel w.r.t. the training dataset. Nevertheless, existing works do not reckon the novelty of the test samples and treat them all equally for evaluating generalization. Such novelty-based evaluations are of significance to validate the fitness of a CNN in autonomous driving applications. Hence, we propose a CNN generalization scoring framework that considers novelty of objects in the test dataset. We begin with the representation learning technique to reduce the image data into a low-dimensional space. It is on this space we estimate the novelty of the test samples. Finally, we calculate the generalization score as a combination of the test data prediction performance and novelty. We perform an experimental study of the same for our traffic light detection application. In addition, we systematically visualize the results for an interpretable notion of novelty.

[1]  Hans-Peter Kriegel,et al.  LOF: identifying density-based local outliers , 2000, SIGMOD '00.

[2]  Daniel Cremers,et al.  q-Space Novelty Detection with Variational Autoencoders , 2018, Computational Diffusion MRI.

[3]  Arvind Kumar Shekar,et al.  Label-Free Robustness Estimation of Object Detection CNNs for Autonomous Driving Applications , 2021, International Journal of Computer Vision.

[4]  Karsten Behrendt,et al.  A deep learning approach to traffic lights: Detection, tracking, and classification , 2017, 2017 IEEE International Conference on Robotics and Automation (ICRA).

[5]  Bernhard Schölkopf,et al.  Support Vector Method for Novelty Detection , 1999, NIPS.

[6]  Yue Zhao,et al.  PyOD: A Python Toolbox for Scalable Outlier Detection , 2019, J. Mach. Learn. Res..

[7]  Luigi Palopoli,et al.  Outlier detection for simple default theories , 2010, Artif. Intell..

[8]  Dubravko Miljkovic,et al.  Review of novelty detection methods , 2010, The 33rd International Convention MIPRO.

[9]  Andreas Fregin,et al.  The DriveU Traffic Light Dataset: Introduction and Comparison with Existing Datasets , 2018, 2018 IEEE International Conference on Robotics and Automation (ICRA).

[10]  Tomás Pevný,et al.  Loda: Lightweight on-line detector of anomalies , 2016, Machine Learning.

[11]  Mathias Johanson,et al.  Big Automotive Data: Leveraging large volumes of data for knowledge-driven product development , 2014, 2014 IEEE International Conference on Big Data (Big Data).

[12]  Qinghe Zheng,et al.  Improvement of Generalization Ability of Deep CNN via Implicit Regularization in Two-Stage Training Process , 2018, IEEE Access.

[13]  Maurizio Filippone,et al.  A comparative evaluation of novelty detection algorithms for discrete sequences , 2019, Artificial Intelligence Review.

[14]  B. Lerner,et al.  Online Cluster Drift Detection for Novelty Detection in Data Streams , 2020, 2020 19th IEEE International Conference on Machine Learning and Applications (ICMLA).

[15]  Bhavani M. Thuraisingham,et al.  Classification and Novel Class Detection in Concept-Drifting Data Streams under Time Constraints , 2011, IEEE Transactions on Knowledge and Data Engineering.

[16]  Liang Gou,et al.  VATLD: A Visual Analytics System to Assess, Understand and Improve Traffic Light Detection , 2020, IEEE Transactions on Visualization and Computer Graphics.

[17]  Guillaume Desjardins,et al.  Understanding disentangling in β-VAE , 2018, ArXiv.

[18]  H. Lian On feature selection with principal component analysis for one-class SVM , 2012, Pattern Recognit. Lett..

[19]  Sameer Singh,et al.  Novelty detection: a review - part 1: statistical approaches , 2003, Signal Process..

[20]  Ammar Belatreche,et al.  An experimental evaluation of novelty detection methods , 2014, Neurocomputing.

[21]  Andreas Dengel,et al.  Histogram-based Outlier Score (HBOS): A fast Unsupervised Anomaly Detection Algorithm , 2012 .

[22]  Dacheng Tao,et al.  Control Batch Size and Learning Rate to Generalize Well: Theoretical and Empirical Evidence , 2019, NeurIPS.

[23]  M. Delgado,et al.  Novelty detection methodology based on multi-modal one-class support vector machine , 2015, 2015 IEEE 10th International Symposium on Diagnostics for Electrical Machines, Power Electronics and Drives (SDEMPED).

[24]  Kai Ming Ting,et al.  Fast Anomaly Detection for Streaming Data , 2011, IJCAI.

[25]  Bernd Bischl,et al.  Robust Anomaly Detection in Images using Adversarial Autoencoders , 2019, ECML/PKDD.

[26]  Bokyoung Kang,et al.  Novelty-focused patent mapping for technology opportunity analysis , 2015 .

[27]  Zhi-Hua Zhou,et al.  Isolation Forest , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[28]  Leland McInnes,et al.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction , 2018, ArXiv.

[29]  Bu-Sung Lee,et al.  Autoencoder-based network anomaly detection , 2018, 2018 Wireless Telecommunications Symposium (WTS).

[30]  Michael Teutsch,et al.  Generalization ability of region proposal networks for multispectral person detection , 2019, Automatic Target Recognition XXIX.