Estimating catch rates in real time: Development of a deep learning based Nephrops (Nephrops norvegicus) counter for demersal trawl fisheries

Demersal trawling is largely a blind process where information on catch rates and compositions is only available once the catch is taken onboard the vessel. Obtaining quantitative information on catch rates of target species while fishing can improve a fisheries economic and environmental performance as fishers would be able to use this information to make informed decisions during fishing. Despite there are real-time underwater monitoring systems developed for this purpose, the video data produced by these systems is not analyzed in near real-time. In other words, the user is expected to watch the video feed continuously to evaluate catch rates and composition. This is obviously a demanding process in which quantification of the fish counts will be of a qualitative nature. In this study, underwater footages collected using an in-trawl video recording system were processed to detect, track, and count the number of individuals of the target species, Nephrops norvegicus, entering the trawl in real-time. The detection was accomplished using a You Only Look Once v4 (YOLOv4) algorithm. Two other variants of the YOLOv4 algorithm (tiny and scaled) were included in the study to compare their effects on the accuracy of the subsequent steps and overall speed of the processing. SORT algorithm was used as the tracker and any Nephrops that cross the horizontal level at 4/5 of the frame height were counted as catch. The detection performance of the YOLOv4 model provided a mean average precision (mAP@50) value of 97.82%, which is higher than the other two variants. However, the average processing speed of the tiny model is the highest with 253.51 frames per second. A correct count rate of 80.73% was achieved by YOLOv4 when the total number of Nephrops are considered in all the test videos. In conclusion, this approach was successful in processing underwater images in real time to determine the catch rates of the target species. The approach has great potential to process multiple species simultaneously in order to provide quantitative information not only on the target species but also bycatch and unwanted species to provide a comprehensive picture of the catch composition.

[1]  Md. Rakibul Haque,et al.  YOLO-Fish: A robust fish detection model to detect fish in realistic underwater environment , 2022, Ecol. Informatics.

[2]  D. Vijaya Kumar,et al.  A fast feature selection technique for real-time face detection using hybrid optimized region based convolutional neural network , 2022, Multimedia Tools and Applications.

[3]  Jinkang Wang,et al.  A Novel Attention-Based Lightweight Network for Multiscale Object Detection in Underwater Images , 2022, Journal of Sensors.

[4]  Mairo Leier,et al.  Environmentally adaptive fish or no-fish classification for river video fish counters using high-performance desktop and embedded hardware , 2022, Ecol. Informatics.

[5]  Yanzhen Gu,et al.  A Robust Underwater Multiclass Fish-School Tracking Algorithm , 2022, Remote. Sens..

[6]  Shijia Zhao,et al.  An Improved YOLO Algorithm for Fast and Accurate Underwater Object Detection , 2022, Symmetry.

[7]  L. Krag,et al.  Test and development of a sediment suppressing system for catch monitoring in demersal trawls , 2022, Fisheries Research.

[8]  Q. Yang,et al.  An underwater target recognition method based on improved YOLOv4 in complex marine environment , 2022, Systems Science & Control Engineering.

[9]  Daoliang Li,et al.  Real-time detection and tracking of fish abnormal behavior based on improved YOLOV5 and SiamRPN++ , 2021, Comput. Electron. Agric..

[10]  Ran Tao,et al.  Deep Learning for Unmanned Aerial Vehicle-Based Object Detection and Tracking: A survey , 2021, IEEE Geoscience and Remote Sensing Magazine.

[11]  Wei Song,et al.  Lightweight Underwater Object Detection Based on YOLO v4 and Multi-Scale Attentional Feature Fusion , 2021, Remote. Sens..

[12]  L. Krag,et al.  A Deep Learning Approach to Assist Sustainability of Demersal Trawling Operations , 2021, Sustainability.

[13]  L. Krag,et al.  Towards sustainable demersal fisheries: NepCon image acquisition system for automatic Nephrops norvegicus detection , 2021, PloS one.

[14]  B. Kusy,et al.  Automatic detection of fish and tracking of movement for ecology , 2021, Ecology and evolution.

[15]  Nikos Petrellis,et al.  Measurement of Fish Morphological Features through Image Processing and Deep Learning Techniques , 2021, Applied Sciences.

[16]  Chengquan Zhou,et al.  Real-time nondestructive fish behavior detecting in mixed polyculture system using deep-learning and low-cost devices , 2021, Expert Syst. Appl..

[17]  B. Halpern,et al.  Protecting the global ocean for biodiversity, food and climate , 2021, Nature.

[18]  Ayman Atia,et al.  YOLO fish detection with Euclidean tracking in fish farms , 2021, Journal of Ambient Intelligence and Humanized Computing.

[19]  Quoc V. Le,et al.  Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[20]  Chien-Yao Wang,et al.  Scaled-YOLOv4: Scaling Cross Stage Partial Network , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Philip H. S. Torr,et al.  HOTA: A Higher Order Metric for Evaluating Multi-object Tracking , 2020, International Journal of Computer Vision.

[22]  OUP accepted manuscript , 2021, ICES Journal of Marine Science.

[23]  Yaoguang Wei,et al.  Application of computer vision in fish intelligent feeding system—A review , 2020 .

[24]  Sultan Daud Khan,et al.  Automatic Detection of Nephrops norvegicus Burrows in Underwater Images Using Deep Learning , 2020, 2020 Global Conference on Wireless and Optical Technologies (GCWOT).

[25]  Hong Liu,et al.  Towards Domain Generalization In Underwater Object Detection , 2020, 2020 IEEE International Conference on Image Processing (ICIP).

[26]  Chi-Hsuan Tseng,et al.  Detecting and counting harvested fish and identifying fish types in electronic monitoring system videos using deep convolutional neural networks , 2020, ICES Journal of Marine Science.

[27]  Haitao Zhu,et al.  Underwater Image Processing and Object Detection Based on Deep CNN Method , 2020, J. Sensors.

[28]  Ajmal Mian,et al.  Fish detection and species classification in underwater environments using deep learning with temporal information , 2020, Ecol. Informatics.

[29]  Hong-Yuan Mark Liao,et al.  YOLOv4: Optimal Speed and Accuracy of Object Detection , 2020, ArXiv.

[30]  Mohamed S. Shehata,et al.  Local null space pursuit for real-time moving object detection in aerial surveillance , 2020, Signal Image Video Process..

[31]  Jiucai Jin,et al.  Vision-Based Target Tracking for Unmanned Surface Vehicle Considering Its Motion Features , 2020, IEEE Access.

[32]  Ayman Nabil,et al.  MSR-YOLO: Method to Enhance Fish Detection and Tracking in Fish Farms , 2020, ANT/EDI40.

[33]  Song Hong,et al.  Fast and Accurate Fish Detection Design with Improved YOLO-v3 Model and Transfer Learning , 2020 .

[34]  Coby L. Needle,et al.  Deep neural networks for analysis of fisheries surveillance video and automated monitoring of fish discards , 2020, ICES Journal of Marine Science.

[35]  Ming Zhong,et al.  Application of improved MobileNet-SSD on underwater sea cucumber detection robot , 2019, 2019 IEEE 4th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC).

[36]  Xuefeng Liu,et al.  Real-time Marine Animal Images Classification by Embedded System Based on Mobilenet and Transfer Learning , 2019, OCEANS 2019 - Marseille.

[37]  Quoc V. Le,et al.  Searching for MobileNetV3 , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[38]  Abhishek Dutta,et al.  The VIA Annotation Software for Images, Audio and Video , 2019, ACM Multimedia.

[39]  A. Fernö,et al.  Species-specific residence times in the aft part of a pelagic survey trawl: implications for inference of pre-capture spatial distribution using the Deep Vision system , 2018 .

[40]  Ziqiang Zheng,et al.  Fish Recognition from a Vessel Camera Using Deep Convolutional Neural Network and Data Augmentation , 2018, 2018 OCEANS - MTS/IEEE Kobe Techno-Oceans (OTO).

[41]  Mark Sandler,et al.  MobileNetV2: Inverted Residuals and Linear Bottlenecks , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[42]  R. García,et al.  Real-time fish detection in trawl nets , 2017, OCEANS 2017 - Aberdeen.

[43]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[44]  Dietrich Paulus,et al.  Simple online and realtime tracking with a deep association metric , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[45]  Christopher J. Smith,et al.  The footprint of bottom trawling in European waters: distribution, intensity, and seabed integrity , 2016, ICES Journal of Marine Science.

[46]  Fabio Tozeto Ramos,et al.  Simple online and realtime tracking , 2016, 2016 IEEE International Conference on Image Processing (ICIP).

[47]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[48]  A. Christensen,et al.  The use of at‐sea‐sampling data to dissociate environmental variability in Norway lobster (Nephrops norvegicus) catches to improve resource exploitation efficiency within the Skagerrak/Kattegat trawl fishery , 2015 .

[49]  Shale Rosen,et al.  Deep Vision: An In-Trawl Stereo Camera Makes a Step Forward in Monitoring the Pelagic Community , 2014, PloS one.

[50]  Bruce A. Draper,et al.  Visual object tracking using adaptive correlation filters , 2010, 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition.

[51]  Harold W. Kuhn,et al.  The Hungarian method for the assignment problem , 1955, 50 Years of Integer Programming.

[52]  Jacopo Aguzzi,et al.  A history of recent advancements on Nephrops norvegicus behavioral and physiological rhythms , 2008, Reviews in Fish Biology and Fisheries.

[53]  Sangster.  The Behaviour of the Norway Lobster , Nephrops norvegicus ( L . 1 , During Trawling , 2004 .

[54]  M. Bergmann,et al.  Discard composition of the Nephrops fishery in the Clyde Sea area, Scotland , 2002 .

[55]  O. Tully,et al.  Causes and spatial scales of variability in population structure of Nephrops norvegicus (L.) in the Irish Sea , 1995 .

[56]  R. E. Kalman,et al.  A New Approach to Linear Filtering and Prediction Problems , 2002 .