Deep Neural Networks for Marine Debris Detection in Sonar Images

Garbage and waste disposal is one of the biggest challenges currently faced by mankind. Proper waste disposal and recycling is a must in any sustainable community, and in many coastal areas there is significant water pollution in the form of floating or submerged garbage. This is called marine debris. Submerged marine debris threatens marine life, and for shallow coastal areas, it can also threaten fishing vessels [Iniguez et al. 2016, Renewable and Sustainable Energy Reviews]. Submerged marine debris typically stays in the environment for a long time (20+ years), and consists of materials that can be recycled, such as metals, plastics, glass, etc. Many of these items should not be disposed in water bodies as this has a negative effect in the environment and human health. This thesis performs a comprehensive evaluation on the use of DNNs for the problem of marine debris detection in FLS images, as well as related problems such as image classification, matching, and detection proposals. We do this in a dataset of 2069 FLS images that we captured with an ARIS Explorer 3000 sensor on marine debris objects lying in the floor of a small water tank. The objects we used to produce this dataset contain typical household marine debris and distractor marine objects (tires, hooks, valves, etc), divided in 10 classes plus a background class. Our results show that for the evaluated tasks, DNNs are a superior technique than the corresponding state of the art. There are large gains particularly for the matching and detection proposal tasks. We also study the effect of sample complexity and object size in many tasks, which is valuable information for practitioners. We expect that our results will advance the objective of using Autonomous Underwater Vehicles to automatically survey, detect and collect marine debris from underwater environments.

[1]  David J. C. MacKay,et al.  Information Theory, Inference, and Learning Algorithms , 2004, IEEE Transactions on Information Theory.

[2]  J. Leeuw Multidimensional Scaling Using Majorization : SMACOF in , 2008 .

[3]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Yvan Petillot,et al.  Automated approach to classification of mine-like objects in sidescan sonar using highlight and shadow information , 2004 .

[5]  Barak A. Pearlmutter,et al.  Automatic differentiation in machine learning: a survey , 2015, J. Mach. Learn. Res..

[6]  Matias Valdenegro-Toro Real-time convolutional networks for sonar image classification in low-power embedded systems , 2017, ESANN.

[7]  C. Wilcox,et al.  Plastic waste inputs from land into the ocean , 2015, Science.

[8]  Nikos Komodakis,et al.  Learning to compare image patches via convolutional neural networks , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[9]  Leigh McCue,et al.  Handbook of Marine Craft Hydrodynamics and Motion Control [Bookshelf] , 2016, IEEE Control Systems.

[10]  H. Tse,et al.  Plastic waste in the marine environment: A review of sources, occurrence and effects. , 2016, The Science of the total environment.

[11]  Pietro Perona,et al.  Microsoft COCO: Common Objects in Context , 2014, ECCV.

[12]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[13]  Abdelhak M. Zoubir,et al.  Sparse Representation based Classification for mine hunting using Synthetic Aperture Sonar , 2012, 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[14]  A. Fullana,et al.  Marine debris occurrence and treatment: A review , 2016 .

[15]  Kevin P. Murphy,et al.  Machine learning - a probabilistic perspective , 2012, Adaptive computation and machine learning series.

[16]  Derek Hoiem,et al.  Category Independent Object Proposals , 2010, ECCV.

[17]  C. Lawrence Zitnick,et al.  Edge Boxes: Locating Object Proposals from Edges , 2014, ECCV.

[18]  Qiang Chen,et al.  Network In Network , 2013, ICLR.

[19]  D. W. Laist Impacts of Marine Debris: Entanglement of Marine Life in Marine Debris Including a Comprehensive List of Species with Entanglement and Ingestion Records , 1997 .

[20]  Stefan Carlsson,et al.  CNN Features Off-the-Shelf: An Astounding Baseline for Recognition , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[21]  Aleksander Madry,et al.  How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) , 2018, NeurIPS.

[22]  John J. Leonard,et al.  Past, Present, and Future of Simultaneous Localization and Mapping: Toward the Robust-Perception Age , 2016, IEEE Transactions on Robotics.

[23]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[24]  Bernt Schiele,et al.  What Makes for Effective Detection Proposals? , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Samy Bengio,et al.  Understanding deep learning requires rethinking generalization , 2016, ICLR.

[26]  Chen Sun,et al.  Revisiting Unreasonable Effectiveness of Data in Deep Learning Era , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[27]  Yan Pailhas,et al.  Cascade of Boosted Classifiers for Rapid Detection of Underwater Objects , 2010 .

[28]  Andrew Zisserman,et al.  Spatial Transformer Networks , 2015, NIPS.

[29]  Paul A. Viola,et al.  Rapid object detection using a boosted cascade of simple features , 2001, Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001.

[30]  Geoffrey I. Webb,et al.  Encyclopedia of Machine Learning , 2011, Encyclopedia of Machine Learning.

[31]  Esther Dura Image Processing Techniques For the Detection and Classification of Man Made Objects in Side-Scan Sonar Images , 2011 .

[32]  Roberto Cipolla,et al.  Multi-task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[33]  Jiri Matas,et al.  Systematic evaluation of convolution neural network advances on the Imagenet , 2017, Comput. Vis. Image Underst..

[34]  Massimo Caccia,et al.  Improving Automatic Target Recognition with Forward Looking Sonar Mosaics , 2014 .

[35]  Behzad Kamgar-Parsi,et al.  Underwater imaging with a moving acoustic lens , 1998, IEEE Trans. Image Process..

[36]  Neelima Chavali,et al.  Object-Proposal Evaluation Protocol is ‘Gameable’ , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[37]  D. W. Laist,et al.  Overview of the biological effects of lost and discarded plastic debris in the marine environment , 1987 .

[38]  Sebastian Ruder,et al.  An overview of gradient descent optimization algorithms , 2016, Vestnik komp'iuternykh i informatsionnykh tekhnologii.

[39]  Georg E. Beduhn REMOVAL OF OIL AND DEBRIS FROM HARBOR WATERS , 1966 .

[40]  Xiang Zhang,et al.  OverFeat: Integrated Recognition, Localization and Detection using Convolutional Networks , 2013, ICLR.

[41]  Jamil Sawas,et al.  Cascade of boosted classifiers for automatic target recognition in synthetic aperture sonar imagery , 2013 .

[42]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[43]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[44]  Luc Van Gool,et al.  The Pascal Visual Object Classes (VOC) Challenge , 2010, International Journal of Computer Vision.

[45]  John A. Fawcett,et al.  A Template Matching Procedure for Automatic Target Recognition in Synthetic Aperture Sonar Imagery , 2010, IEEE Signal Processing Letters.

[46]  G. Papatheodorou,et al.  Marine Debris on the Seafloor of the Mediterranean Sea: Examples from Two Enclosed Gulfs in Western Greece , 1999 .

[47]  Matthew B. Blaschko,et al.  Learning a category independent object detection cascade , 2011, 2011 International Conference on Computer Vision.

[48]  Luc Van Gool,et al.  SURF: Speeded Up Robust Features , 2006, ECCV.

[49]  Thomas Deselaers,et al.  Measuring the Objectness of Image Windows , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[50]  C. Rochman Strategies for reducing ocean plastic debris should be diverse and guided by science , 2016 .

[51]  Matthijs C. Dorst Distinctive Image Features from Scale-Invariant Keypoints , 2011 .

[52]  S. Chiba,et al.  Human footprint in the abyss: 30 year records of deep-sea plastic debris , 2018, Marine Policy.

[53]  Nicola Neretti,et al.  Mosaicing of acoustic camera images , 2005 .

[54]  M. Witt,et al.  Marine anthropogenic litter on British beaches: A 10-year nationwide assessment using citizen science data. , 2017, The Science of the total environment.

[55]  Gary R. Bradski,et al.  ORB: An efficient alternative to SIFT or SURF , 2011, 2011 International Conference on Computer Vision.

[56]  Luc Van Gool,et al.  The Pascal Visual Object Classes Challenge: A Retrospective , 2014, International Journal of Computer Vision.

[57]  Dieter Kraus,et al.  Hand-Crafted Feature Based Classification against Convolutional Neural Networks for False Alarm Reduction on Active Diver Detection Sonar Data , 2018, OCEANS 2018 MTS/IEEE Charleston.

[58]  Yoshua Bengio,et al.  Understanding the difficulty of training deep feedforward neural networks , 2010, AISTATS.

[59]  Gabriel Oliver,et al.  Intervention AUVs: The next challenge , 2015, Annu. Rev. Control..

[60]  Nobuhito Mori,et al.  Survey of 2011 Tohoku earthquake tsunami inundation and run‐up , 2011 .

[61]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[62]  Gang Hua,et al.  Discriminative Learning of Local Image Descriptors , 1990, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[63]  S. Negahdaripour,et al.  Dynamic scene analysis and mosaicing of benthic habitats by FS sonar imaging - Issues and complexities , 2011, OCEANS'11 MTS/IEEE KONA.

[64]  David M. Lane,et al.  Object classification with convolution neural network based on the time-frequency representation of their echo , 2017, 2017 IEEE 27th International Workshop on Machine Learning for Signal Processing (MLSP).

[65]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[66]  Wan Nurulhuda Wan Shamsuri,et al.  Design of Rubbish Collecting System for Inland Waterways , 2015 .

[67]  Yann LeCun,et al.  Stereo Matching by Training a Convolutional Neural Network to Compare Image Patches , 2015, J. Mach. Learn. Res..

[68]  Rich Caruana,et al.  Multitask Learning , 1998, Encyclopedia of Machine Learning and Data Mining.

[69]  Julia Reisser,et al.  Plastic Pollution in the World's Oceans: More than 5 Trillion Plastic Pieces Weighing over 250,000 Tons Afloat at Sea , 2014, PloS one.

[70]  Vincent Lepetit,et al.  LIFT: Learned Invariant Feature Transform , 2016, ECCV.

[71]  Matias Valdenegro-Toro Objectness Scoring and Detection Proposals in Forward-Looking Sonar Images with Convolutional Neural Networks , 2016, ANNPR.

[72]  E. O. Belcher,et al.  Beamforming and imaging with acoustic lenses in small, high-frequency sonars , 1999, Oceans '99. MTS/IEEE. Riding the Crest into the 21st Century. Conference and Exhibition. Conference Proceedings (IEEE Cat. No.99CH37008).

[73]  Sebastian Ruder,et al.  An Overview of Multi-Task Learning in Deep Neural Networks , 2017, ArXiv.

[74]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[75]  Zhiqiang Shen,et al.  DSOD: Learning Deeply Supervised Object Detectors from Scratch , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[76]  C. Wilcox,et al.  Differentiating littering, urban runoff and marine transport as sources of marine debris in coastal and estuarine environments , 2017, Scientific Reports.

[77]  Leonidas J. Guibas,et al.  PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[78]  E. Belcher,et al.  Dual-Frequency Identification Sonar (DIDSON) , 2002, Proceedings of the 2002 Interntional Symposium on Underwater Technology (Cat. No.02EX556).

[79]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[80]  Rafael C. González,et al.  Digital image processing, 3rd Edition , 2008 .

[81]  Joaquim Salvi,et al.  Automatic detection of underwater chain links using a forward-looking sonar , 2013, 2013 MTS/IEEE OCEANS - Bergen.

[82]  E. O. Belcher,et al.  Object identification with acoustic lenses , 2001, MTS/IEEE Oceans 2001. An Ocean Odyssey. Conference Proceedings (IEEE Cat. No.01CH37295).

[83]  K. Fink Computer Simulation of Pressure Fields Generated by Acoustic Lens Beamformers , 1994 .

[84]  Ping Luo,et al.  Towards Understanding Regularization in Batch Normalization , 2018, ICLR.

[85]  Rachid Deriche,et al.  A Robust Technique for Matching two Uncalibrated Images Through the Recovery of the Unknown Epipolar Geometry , 1995, Artif. Intell..

[86]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[87]  M. Costa,et al.  Methods applied in studies of benthic marine debris. , 2008, Marine pollution bulletin.

[88]  Yann LeCun,et al.  Signature Verification Using A "Siamese" Time Delay Neural Network , 1993, Int. J. Pattern Recognit. Artif. Intell..

[89]  Andreas Birk,et al.  Seafloor classification for mine countermeasures operations using synthetic aperture sonar images , 2017, OCEANS 2017 - Aberdeen.

[90]  CONVOLUTIONAL NEURAL NETWORK TRANSFER LEARNING FOR UNDERWATER OBJECT CLASSIFICATION , 2018 .

[91]  Sanjiv Kumar,et al.  On the Convergence of Adam and Beyond , 2018 .

[92]  Bo Fu,et al.  Deep learning feature extraction for target recognition and classification in underwater sonar images , 2017, 2017 IEEE 56th Annual Conference on Decision and Control (CDC).

[93]  Robert J. Hofman,et al.  Environmental effects of marine fishing , 1995 .

[94]  Isabelle Guyon,et al.  Taking Human out of Learning Applications: A Survey on Automated Machine Learning , 2018, 1810.13306.

[95]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[96]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[97]  S. B. Sheavly,et al.  Marine Debris & Plastics: Environmental Concerns, Sources, Impacts and Solutions , 2007 .

[98]  Risto Miikkulainen,et al.  Evolving Neural Networks through Augmenting Topologies , 2002, Evolutionary Computation.

[99]  Yoshua Bengio,et al.  Deep Sparse Rectifier Neural Networks , 2011, AISTATS.

[100]  P. Ryan Litter survey detects the South Atlantic 'garbage patch'. , 2014, Marine pollution bulletin.

[101]  Daniel Pauly,et al.  Population Trend of the World’s Monitored Seabirds, 1950-2010 , 2015, PloS one.

[102]  A. Vardy,et al.  Side-scan sonar image registration for AUV navigation , 2010, 2011 IEEE Symposium on Underwater Technology and Workshop on Scientific Use of Submarine Cables and Related Technologies.

[103]  Forrest N. Iandola,et al.  SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <1MB model size , 2016, ArXiv.

[104]  Xinjiang Wang,et al.  Understanding Regularization in Batch Normalization , 2018 .

[105]  Jitendra Malik,et al.  DeepBox: Learning Objectness with Convolutional Networks , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[106]  Matias Valdenegro-Toro,et al.  Best practices in convolutional networks for forward-looking sonar image recognition , 2017, OCEANS 2017 - Aberdeen.

[107]  Bernt Schiele,et al.  How good are detection proposals, really? , 2014, BMVC.

[108]  B. D. Hardesty,et al.  Threat of plastic pollution to seabirds is global, pervasive, and increasing , 2015, Proceedings of the National Academy of Sciences.

[109]  Thomas Bräunl,et al.  Review of underwater SLAM techniques , 2015, 2015 6th International Conference on Automation, Robotics and Applications (ICARA).

[110]  Joaquim Salvi,et al.  Real-time mosaicing with two-dimensional forward-looking sonar , 2014, 2014 IEEE International Conference on Robotics and Automation (ICRA).

[111]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[112]  Ross B. Girshick,et al.  Fast R-CNN , 2015, 1504.08083.

[113]  Didier Gueriot,et al.  Guided block-matching for sonar image registration using unsupervised Kohonen neural networks , 2013, 2013 OCEANS - San Diego.

[114]  Yarin Gal,et al.  Uncertainty in Deep Learning , 2016 .

[115]  Aleksander Madry,et al.  How Does Batch Normalization Help Optimization? (No, It Is Not About Internal Covariate Shift) , 2018, NIPS 2018.

[116]  Yoram Singer,et al.  Adaptive Subgradient Methods for Online Learning and Stochastic Optimization , 2011, J. Mach. Learn. Res..

[117]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[118]  James Philbin,et al.  FaceNet: A unified embedding for face recognition and clustering , 2015, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[120]  Koen E. A. van de Sande,et al.  Selective Search for Object Recognition , 2013, International Journal of Computer Vision.

[121]  Ales Leonardis,et al.  Visual Object Tracking Performance Measures Revisited , 2015, IEEE Transactions on Image Processing.

[122]  Silvio Savarese,et al.  Learning to Track at 100 FPS with Deep Regression Networks , 2016, ECCV.

[123]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[124]  Philip H. S. Torr,et al.  BING: Binarized normed gradients for objectness estimation at 300fps , 2019, Computational Visual Media.

[125]  Marine Debris, Beach Quality, and Non-Market Values , 1996 .

[126]  Richard C. Thompson,et al.  The impact of debris on marine life. , 2015, Marine pollution bulletin.

[127]  David P. Williams Underwater target classification in synthetic aperture sonar imagery using deep convolutional neural networks , 2016, 2016 23rd International Conference on Pattern Recognition (ICPR).

[128]  Christopher Barngrover,et al.  Semisynthetic Versus Real-World Sonar Training Data for the Classification of Mine-Like Objects , 2015, IEEE Journal of Oceanic Engineering.

[129]  Vincent Lepetit,et al.  A fast local descriptor for dense matching , 2008, 2008 IEEE Conference on Computer Vision and Pattern Recognition.

[130]  Patrick Mair,et al.  Multidimensional Scaling Using Majorization: SMACOF in R , 2008 .

[131]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[132]  Yan Pailhas,et al.  High-Resolution Sonars: What Resolution Do We Need for Target Recognition? , 2010, EURASIP J. Adv. Signal Process..

[133]  Jian Sun,et al.  Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition , 2014, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[134]  Surya Ganguli,et al.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks , 2013, ICLR.

[135]  Judith L. Connor,et al.  Debris in the deep: Using a 22-year video annotation database to survey marine litter in Monterey Canyon, central California, USA , 2013 .

[136]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[137]  Dumitru Erhan,et al.  Scalable Object Detection Using Deep Neural Networks , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[138]  W. J. Kirkwood,et al.  Development of the DORADO mapping vehicle for multibeam, subbottom, and sidescan science missions , 2007, J. Field Robotics.

[139]  Henrik Aanæs,et al.  Interesting Interest Points , 2011, International Journal of Computer Vision.

[140]  Abdelhak M. Zoubir,et al.  Optimal Feature Set for Automatic Detection and Classification of Underwater Objects in SAS Images , 2011, IEEE Journal of Selected Topics in Signal Processing.

[141]  Richard E Engler,et al.  The complex interaction between marine debris and toxic chemicals in the ocean. , 2012, Environmental science & technology.

[142]  Richard Szeliski,et al.  Computer Vision - Algorithms and Applications , 2011, Texts in Computer Science.

[143]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[144]  Roy Edgar Hansen Introduction to sonar , 2009 .