Reliability of GAN Generated Data to Train and Validate Perception Systems for Autonomous Vehicles

Autonomous systems deployed in the real world have to deal with potential problem causing situations that they have never seen during their training phases. Due to the long-tail nature of events, collecting a large amount of data for such corner cases is a difficult task. While simulation is one plausible solution, recent developments in the field of Generative Adversarial Networks (GANs) make them a promising tool to generate and augment realistic data without exhibiting a domain shift from actual real data. In this manuscript, we empirically analyze and propose novel solutions for the trust that we can place on GAN generated data for training and validation of vision-based perception modules like object detection and scenario classification.

[1]  Sepp Hochreiter,et al.  GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium , 2017, NIPS.

[2]  Trevor Darrell,et al.  Long-term recurrent convolutional networks for visual recognition and description , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[3]  Sarfraz Khurshid,et al.  DeepRoad: GAN-based Metamorphic Autonomous Driving System Testing , 2018, ArXiv.

[4]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[5]  Jaakko Lehtinen,et al.  Improved Precision and Recall Metric for Assessing Generative Models , 2019, NeurIPS.

[6]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[7]  Sailik Sengupta,et al.  Imagining an Engineer: On GAN-Based Data Augmentation Perpetuating Biases , 2018, ArXiv.

[8]  Simon Osindero,et al.  Conditional Generative Adversarial Nets , 2014, ArXiv.

[9]  Oriol Vinyals,et al.  Seeing is Not Necessarily Believing: Limitations of BigGANs for Data Augmentation , 2019 .

[10]  Cordelia Schmid,et al.  How good is my GAN? , 2018, ECCV.

[11]  Ke Yan,et al.  Data augmentation using generative adversarial networks (CycleGAN) to improve generalizability in CT segmentation tasks , 2019, Scientific Reports.

[12]  Sukhendu Das,et al.  What’s There in the Dark , 2019, 2019 IEEE International Conference on Image Processing (ICIP).

[13]  Aaron C. Courville,et al.  Improved Training of Wasserstein GANs , 2017, NIPS.

[14]  Jeff Donahue,et al.  Large Scale GAN Training for High Fidelity Natural Image Synthesis , 2018, ICLR.

[15]  Nicu Sebe,et al.  Cross-Domain Car Detection Using Unsupervised Image-to-Image Translation: From Day to Night , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[16]  Jan Kautz,et al.  Context-aware Synthesis and Placement of Object Instances , 2018, NeurIPS.

[17]  Seunghoon Hong,et al.  Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[18]  Jan Kautz,et al.  Multimodal Unsupervised Image-to-Image Translation , 2018, ECCV.

[19]  Masaaki Ikehara,et al.  GAN-Based Rain Noise Removal From Single-Image Considering Rain Composite Models , 2020, IEEE Access.

[20]  Peter Kontschieder,et al.  The Mapillary Vistas Dataset for Semantic Understanding of Street Scenes , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[21]  Thomas S. Huang,et al.  Generative Image Inpainting with Contextual Attention , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[22]  拓海 杉山,et al.  “Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks”の学習報告 , 2017 .

[23]  Jan Kautz,et al.  Video-to-Video Synthesis , 2018, NeurIPS.

[24]  Claus Aranha,et al.  Data Augmentation Using GANs , 2019, ArXiv.

[25]  Trevor Darrell,et al.  BDD100K: A Diverse Driving Video Database with Scalable Annotation Tooling , 2018, ArXiv.

[26]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[27]  Ali Borji,et al.  Pros and Cons of GAN Evaluation Measures , 2018, Comput. Vis. Image Underst..

[28]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[29]  Kilian Q. Weinberger,et al.  An empirical study on evaluation metrics of generative adversarial networks , 2018, ArXiv.

[30]  David Hurych,et al.  Yes, we GAN: Applying Adversarial Techniques for Autonomous Driving , 2019, Autonomous Vehicles and Machines.

[31]  Ross B. Girshick,et al.  Mask R-CNN , 2017, 1703.06870.

[32]  Pratik Prabhanjan Brahma,et al.  Subset Replay Based Continual Learning for Scalable Improvement of Autonomous Systems , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[33]  Alexei A. Efros,et al.  Image-to-Image Translation with Conditional Adversarial Networks , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[34]  Jan Kautz,et al.  High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[35]  George Papandreou,et al.  Rethinking Atrous Convolution for Semantic Image Segmentation , 2017, ArXiv.