Hallucination In Object Detection — A Study In Visual Part VERIFICATION

We show that object detectors can hallucinate and detect missing objects; potentially even accurately localized at their expected, but non-existing, position. This is particularly problematic for applications that rely on visual part verification: detecting if an object part is present or absent. We show how popular object detectors hallucinate objects in a visual part verification task and introduce the first visual part verification dataset: DelftBikes, which has 10,000 bike photographs, with 22 densely annotated parts per image, where some parts may be missing. We explicitly annotated an extra object state label for each part to reflect if a part is missing or intact. We propose to evaluate visual part verification by relying on recall and compare popular object detectors on DelftBikes.

[1]  Hamdi Ben Abdallah,et al.  Automatic Inspection of Aeronautical Mechanical Assemblies by Matching the 3D CAD Model and Real 2D Images , 2019, J. Imaging.

[2]  H. C. Garcia,et al.  Automated Refinement of Automated Visual Inspection Algorithms , 2009, IEEE Transactions on Automation Science and Engineering.

[3]  Ettore Stella,et al.  A Real-Time Visual Inspection System for Railway Maintenance: Automatic Hexagonal-Headed Bolts Detection , 2007, IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews).

[4]  Dusan Koniar,et al.  Virtual Instrumentation for Visual Inspection in Mechatronic Applications , 2014 .

[5]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[6]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[7]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[9]  Patrick S. K. Chua,et al.  Recognition of features of parts subjected to motion using ARTMAP incorporated in a flexible vibratory bowl feeder system , 2006, Artificial Intelligence for Engineering Design, Analysis and Manufacturing.

[10]  Ohad Ben-Shahar,et al.  Exploring the Bounds of the Utility of Context for Object Detection , 2017, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  J. V. Gemert,et al.  On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[12]  Nancy Chinchor,et al.  MUC-4 evaluation metrics , 1992, MUC.

[13]  Yong Jae Lee,et al.  Don’t Judge an Object by Its Context: Learning to Overcome Contextual Bias , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[14]  Young Shik Moon,et al.  Visual inspection system for the classification of solder joints , 1999, Pattern Recognit..

[15]  Albert-Jan Baerveldt.  A vision system for object verification and localization based on local features , 2001, Robotics Auton. Syst..

[16]  I. Andreadis,et al.  Automated Visual Inspection Systems , 2002 .

[17]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[18]  Shiguang Shan,et al.  Structure Inference Net: Object Detection Using Scene-Level Context and Instance-Level Relationships , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[19]  Sanja Fidler,et al.  Detect What You Can: Detecting and Representing Objects Using Holistic Models and Body Parts , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[20]  Björn W. Schuller,et al.  Go-CaRD - Generic, Optical Car Part Recognition and Detection: Collection, Insights, and Applications , 2020, ArXiv.

[21]  Nikos Komodakis,et al.  Object Detection via a Multi-region and Semantic Segmentation-Aware CNN Model , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[22]  Alessio Del Bue,et al.  Automatic inspection of aeronautic components , 2017, Machine Vision and Applications.

[23]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Hanqing Lu,et al.  CoupleNet: Coupling Global Structure with Local Parts for Object Detection , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[25]  Yu Wang,et al.  Shift Equivariance in Object Detection , 2020, ECCV Workshops.

[26]  Narendra Ahuja,et al.  Automated Visual Inspection of Railroad Tracks , 2013, IEEE Transactions on Intelligent Transportation Systems.