Deep learning for smart fish farming: applications, opportunities and challenges

With the rapid emergence of deep learning (DL) technology, it has been successfully used in various fields including aquaculture. This change can create new opportunities and a series of challenges for information and data processing in smart fish farming. This paper focuses on the applications of DL in aquaculture, including live fish identification, species classification, behavioral analysis, feeding decision-making, size or biomass estimation, water quality prediction. In addition, the technical details of DL methods applied to smart fish farming are also analyzed, including data, algorithms, computing power, and performance. The results of this review show that the most significant contribution of DL is the ability to automatically extract features. However, challenges still exist; DL is still in an era of weak artificial intelligence. A large number of labeled data are needed for training, which has become a bottleneck restricting further DL applications in aquaculture. Nevertheless, DL still offers breakthroughs in the handling of complex data in aquaculture. In brief, our purpose is to provide researchers and practitioners with a better understanding of the current state of the art of DL in aquaculture, which can provide strong support for the implementation of smart fish farming.

[1]  M. S. Weltersbach,et al.  Digital camera monitoring of recreational fishing effort: Applications and challenges , 2019, Fish and Fisheries.

[2]  Byunghan Lee,et al.  Deep learning in bioinformatics , 2016, Briefings Bioinform..

[3]  Xi En Cheng,et al.  Zebrafish tracking using convolutional neural networks , 2017, Scientific Reports.

[4]  Emili García-Berthou,et al.  Herbivory and seasonal changes in diet of a highly endemic cyprinodontid fish (Aphanius farsicus) , 2015, Environmental Biology of Fishes.

[5]  Peng Hao,et al.  Transfer learning using computational intelligence: A survey , 2015, Knowl. Based Syst..

[6]  R. Hilborn,et al.  Fisheries management impacts on target species status , 2016, Proceedings of the National Academy of Sciences.

[7]  Marc Chaumont,et al.  A Deep learning method for accurate and fast identification of coral reef fishes in underwater images , 2018, Ecol. Informatics.

[8]  Marcel Simon,et al.  Croatian Fish Dataset: Fine-grained classification of fish species in their natural habitat , 2015 .

[9]  Qi Tian,et al.  SIFT Meets CNN: A Decade Survey of Instance Retrieval , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[10]  Khawar Khurshid,et al.  Automatic fish detection in underwater videos by a deep neural network-based hybrid motion learning system , 2019, ICES Journal of Marine Science.

[11]  Sachit Butail,et al.  Three-dimensional reconstruction of the fast-start swimming kinematics of densely schooling fish , 2012, Journal of The Royal Society Interface.

[12]  Nils Olav Handegard,et al.  Fish species identification using a convolutional neural network trained on synthetic data , 2018, ICES Journal of Marine Science.

[13]  Patrizia Busato,et al.  Machine Learning in Agriculture: A Review , 2018, Sensors.

[14]  Jose Luis Lisani,et al.  Image-based, unsupervised estimation of fish size from commercial landings using deep learning , 2020 .

[15]  Yi-Fan Zhang,et al.  Shrimp recognition using ShrimpNet based on convolutional neural network , 2020 .

[16]  Daming Xu,et al.  An adaptive image enhancement method for a recirculating aquaculture system , 2017, Scientific Reports.

[17]  Jürgen Schmidhuber,et al.  Deep learning in neural networks: An overview , 2014, Neural Networks.

[18]  Bastiaan Star,et al.  Unlocking the potential of ancient fish DNA in the genomic era , 2019, Evolutionary applications.

[19]  Haiyong Zheng,et al.  Improving Transfer Learning and Squeeze- and-Excitation Networks for Small-Scale Fine-Grained Fish Image Classification , 2018, IEEE Access.

[20]  Guang-Zhong Yang,et al.  Deep Learning for Health Informatics , 2017, IEEE Journal of Biomedical and Health Informatics.

[21]  A. Pérez-Escudero,et al.  idTracker: tracking individuals in a group by automatic identification of unmarked animals , 2014, Nature Methods.

[22]  Ajmal Mian,et al.  Fish species classification in unconstrained underwater environments based on deep learning , 2016 .

[23]  Nils Olav Handegard,et al.  Automatic interpretation of otoliths using deep learning , 2018, bioRxiv.

[24]  Hassan Zargarzadeh,et al.  Design and Implementation of an Assistive Real-Time Red Lionfish Detection System for AUV/ROVs , 2018, Complex..

[25]  S. Lester,et al.  Interactions and management for the future of marine aquaculture and capture fisheries , 2019, Fish and Fisheries.

[26]  Amin Taheri-Garavand,et al.  Deep learning-based appearance features extraction for automated carp species identification , 2020 .

[27]  Matt Merrifield,et al.  Opportunities to improve fisheries management through innovative technology and advanced data systems , 2019, Fish and Fisheries.

[28]  Yonggang Wen,et al.  Toward Scalable Systems for Big Data Analytics: A Technology Tutorial , 2014, IEEE Access.

[29]  Quanshi Zhang,et al.  Visual interpretability for deep learning: a survey , 2018, Frontiers of Information Technology & Electronic Engineering.

[30]  Chao Zhou,et al.  Automatic Fish Population Counting by Machine Vision and a Hybrid Deep Neural Network Model , 2020, Animals : an open access journal from MDPI.

[31]  Mukesh Tripathi,et al.  A role of computer vision in fruits and vegetables among various horticulture products of agriculture fields: A survey , 2020 .

[32]  Daoliang Li,et al.  A hybrid WA-CPSO-LSSVR model for dissolved oxygen content prediction in crab culture , 2014, Eng. Appl. Artif. Intell..

[33]  Ariadne Barbosa Gonçalves,et al.  Automatic live fingerlings counting using computer vision , 2019, Comput. Electron. Agric..

[34]  Kyunghyun Cho,et al.  A Comparison of Audio Signal Preprocessing Methods for Deep Neural Networks on Music Tagging , 2017, 2018 26th European Signal Processing Conference (EUSIPCO).

[35]  Mohd Salman Leong,et al.  Challenges and Opportunities of Deep Learning Models for Machinery Fault Detection and Diagnosis: A Review , 2019, IEEE Access.

[36]  Rafael Garcia,et al.  Automatic segmentation of fish using deep learning with application to fish size measurement , 2020, ICES Journal of Marine Science.

[37]  C. Hsieh,et al.  Automatic measurement of the body length of harvested fish using convolutional neural networks , 2020, Biosystems Engineering.

[38]  Deqin Xiao,et al.  Feeding behavior recognition for group-housed pigs with the Faster R-CNN , 2018, Comput. Electron. Agric..

[39]  J. Benzie,et al.  Improving feed efficiency in fish using selective breeding: a review , 2018 .

[40]  Lan Chen,et al.  Near infrared computer vision and neuro-fuzzy model-based feeding decision system for fish in aquaculture , 2018, Comput. Electron. Agric..

[41]  Heng Tao Shen,et al.  Principal Component Analysis , 2009, Encyclopedia of Biometrics.

[42]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[43]  Jean-Marc Odobez,et al.  Multi-Layer Background Subtraction Based on Color and Texture , 2007, 2007 IEEE Conference on Computer Vision and Pattern Recognition.

[44]  Pierre Vandergheynst,et al.  Geometric Deep Learning: Going beyond Euclidean data , 2016, IEEE Signal Process. Mag..

[45]  Mattia G. Bergomi,et al.  idtracker.ai: tracking all individuals in small or large collectives of unmarked animals , 2019, Nature Methods.

[46]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[47]  Lin Meng,et al.  Underwater-Drone With Panoramic Camera for Automatic Fish Recognition Based on Deep Learning , 2018, IEEE Access.

[48]  Qiang Wang,et al.  Benchmarking State-of-the-Art Deep Learning Software Tools , 2016, 2016 7th International Conference on Cloud Computing and Big Data (CCBD).

[49]  Koji Tsuda,et al.  Integration of sonar and optical camera images using deep neural network for fish monitoring , 2019, Aquacultural Engineering.

[50]  Xinting Yang,et al.  Intelligent feeding control methods in aquaculture with an emphasis on fish: a review , 2018 .

[51]  Huimin Lu,et al.  Brain Intelligence: Go beyond Artificial Intelligence , 2017, Mobile Networks and Applications.

[52]  Ajmal Mian,et al.  Fish detection and species classification in underwater environments using deep learning with temporal information , 2020, Ecol. Informatics.

[53]  ZhangGuangquan,et al.  Transfer learning using computational intelligence , 2015 .

[54]  T. Poggio,et al.  Deep vs. shallow networks : An approximation theory perspective , 2016, ArXiv.

[55]  Lizhuang Ma,et al.  Not All Areas Are Equal: Transfer Learning for Semantic Segmentation via Hierarchical Region Selection , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[56]  Elad Osherov,et al.  Automated Analysis of Marine Video with Limited Data , 2018, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[57]  Nicolas Le Roux,et al.  Representational Power of Restricted Boltzmann Machines and Deep Belief Networks , 2008, Neural Computation.

[58]  Dinesh Kumar Vishwakarma,et al.  A review of state-of-the-art techniques for abnormal human activity recognition , 2019, Eng. Appl. Artif. Intell..

[59]  Qian Zhang,et al.  Attention-based recurrent neural networks for accurate short-term and long-term dissolved oxygen prediction , 2019, Comput. Electron. Agric..

[60]  SunMin,et al.  Models for estimating feed intake in aquaculture , 2016 .

[61]  Nurgun Erdol,et al.  Automatic classification of grouper species by their sounds using deep neural networks. , 2018, The Journal of the Acoustical Society of America.

[62]  Alexios Glaropoulos,et al.  A computer-vision system and methodology for the analysis of fish behavior , 2012 .

[63]  Juntao Liu,et al.  A Water Quality Prediction Method Based on the Deep LSTM Network Considering Correlation in Smart Mariculture , 2019, Sensors.

[64]  Seyed-Mohsen Moosavi-Dezfooli,et al.  DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[65]  Ying Liu,et al.  Modified motion influence map and recurrent neural network-based monitoring of the local unusual behaviors for fish school in intensive aquaculture. , 2018 .

[66]  Syed Jawad Hussain Shah,et al.  Visual features based automated identification of fish species using deep convolutional neural networks , 2019, Comput. Electron. Agric..

[67]  Chen Xu,et al.  Detection and Analysis of Behavior Trajectory for Sea Cucumbers Based on Deep Learning , 2020, IEEE Access.

[68]  Changshui Zhang,et al.  DeepFish: Accurate underwater live fish recognition with a deep architecture , 2016, Neurocomputing.

[69]  Alfonso B. Labao,et al.  Cascaded deep network systems with linked ensemble components for underwater fish detection in the wild , 2019, Ecol. Informatics.

[70]  Daoliang Li,et al.  Fish species classification by color, texture and multi-class support vector machine using computer vision , 2012 .

[71]  Miquel Palmer,et al.  Using stereoscopic video cameras to evaluate seagrass meadows nursery function in the Mediterranean , 2017 .

[72]  Dean Zhao,et al.  Real-time robust detector for underwater live crabs based on deep learning , 2020, Comput. Electron. Agric..

[73]  Petr Císar,et al.  Automated within tank fish mass estimation using infrared reflection system , 2018, Comput. Electron. Agric..

[74]  B. Mao,et al.  Subsurface velocity inversion from deep learning-based data assimilation , 2019, Journal of Applied Geophysics.

[76]  Rafael Rieder,et al.  Computer vision and artificial intelligence in precision agriculture for grain crops: A systematic review , 2018, Comput. Electron. Agric..

[77]  Subhashini Venugopalan,et al.  Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. , 2016, JAMA.

[78]  Simone Marini,et al.  Tracking Fish Abundance by Underwater Image Recognition , 2018, Scientific Reports.

[79]  Paul Geladi,et al.  Principal Component Analysis , 1987, Comprehensive Chemometrics.

[80]  Xinting Yang,et al.  Evaluation of fish feeding intensity in aquaculture using a convolutional neural network and machine vision , 2019, Aquaculture.

[81]  Ali Danandeh Mehr,et al.  A comparative analysis among computational intelligence techniques for dissolved oxygen prediction in Delaware River , 2017 .

[82]  Ashfaqur Rahman,et al.  Dissolved oxygen prediction in prawn ponds from a group of one step predictors , 2020 .

[83]  Tao Wang,et al.  End-to-end text recognition with convolutional neural networks , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[84]  Arthur F. A. Fernandes,et al.  Deep Learning image segmentation for extraction of fish body measurements and prediction of body weight and carcass traits in Nile tilapia , 2020, Comput. Electron. Agric..

[85]  James Harle,et al.  Can marine fisheries and aquaculture meet fish demand from a growing human population in a changing climate , 2012 .

[86]  Lingxi Peng,et al.  An intelligent aerator algorithm inspired-by deep learning. , 2019, Mathematical biosciences and engineering : MBE.

[87]  Agnar Aamodt,et al.  A spatio-temporal recurrent network for salmon feeding action recognition from underwater videos in aquaculture , 2019, Comput. Electron. Agric..

[88]  J. Ross Quinlan,et al.  Induction of Decision Trees , 1986, Machine Learning.

[89]  Lan Chen,et al.  Handling Water Reflections for Computer Vision in Aquaculture , 2018 .

[90]  SchmidhuberJürgen.  Deep learning in neural networks , 2015 .

[91]  Nikhil Ketkar,et al.  Introduction to PyTorch , 2021, Deep Learning with Python.

[92]  Dazhong Wu,et al.  Deep learning for smart manufacturing: Methods and applications , 2018, Journal of Manufacturing Systems.

[93]  Bradley J. Erickson,et al.  Toolkits and Libraries for Deep Learning , 2017, Journal of Digital Imaging.

[94]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[95]  Ying Liu,et al.  Semi-supervised learning-based live fish identification in aquaculture using modified deep convolutional generative adversarial networks , 2018 .

[96]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[97]  Loic A. Royer,et al.  Applications, Promises, and Pitfalls of Deep Learning for Fluorescence Image Reconstruction , 2018 .

[98]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[99]  Yan Qiu Chen,et al.  Robust tracking of fish schools using CNN for head identification , 2017, Multimedia Tools and Applications.

[100]  Dong An,et al.  Research of dissolved oxygen prediction in recirculating aquaculture systems based on deep belief network , 2020 .

[101]  Robert B. Fisher,et al.  Supporting ground-truth annotation of image datasets using clustering , 2012, Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012).

[103]  Giancarlo Marafioti,et al.  Modelling growth performance and feeding behaviour of Atlantic salmon (Salmo salar L.) in commercial-size aquaculture net pens: Model details and validation through full-scale experiments , 2016, Aquaculture.

[104]  T. Poggio,et al.  The Mathematics of Learning: Dealing with Data , 2005, 2005 International Conference on Neural Networks and Brain.

[105]  Bram van Ginneken,et al.  A survey on deep learning in medical image analysis , 2017, Medical Image Anal..

[106]  Li Fei-Fei,et al.  ImageNet: A large-scale hierarchical image database , 2009, CVPR.

[107]  Daoliang Li,et al.  Nonintrusive methods for biomass estimation in aquaculture with emphasis on fish: a review , 2020, Reviews in Aquaculture.

[108]  Hyun Myung,et al.  Image-Based Monitoring of Jellyfish Using Deep Learning Architecture , 2016, IEEE Sensors Journal.

[109]  Md. Sumon Shahriar,et al.  A Dynamic Data-driven Decision Support for Aquaculture Farm Closure , 2014, ICCS.

[110]  W. Watson,et al.  The behavior of cod (Gadus morhua) in an offshore aquaculture net pen , 2011 .

[111]  Taghi M. Khoshgoftaar,et al.  Deep learning applications and challenges in big data analytics , 2015, Journal of Big Data.

[112]  Jan Urban,et al.  Application of machine vision systems in aquaculture with emphasis on fish: state-of-the-art and key issues , 2017 .

[113]  Noel D.G. White,et al.  Feasibility of near-infrared hyperspectral imaging to differentiate Canadian wheat classes , 2008 .

[114]  Norval J. C. Strachan,et al.  Automated measurement of species and length of fish by computer vision , 2006 .

[115]  Junyu Dong,et al.  Transferring deep knowledge for object recognition in Low-quality underwater videos , 2018, Neurocomputing.

[116]  Allan Hanbury,et al.  A survey of methods for image annotation , 2008, J. Vis. Lang. Comput..

[117]  Wesley Nunes Gonçalves,et al.  Improving Pantanal fish species recognition through taxonomic ranks in convolutional neural networks , 2019, Ecol. Informatics.

[118]  Boaz Zion,et al.  Review: The use of computer vision technologies in aquaculture - A review , 2012 .

[119]  Min Sun,et al.  Models for estimating feed intake in aquaculture: A review , 2016, Comput. Electron. Agric..

[120]  Andreas Kamilaris,et al.  Deep learning in agriculture: A survey , 2018, Comput. Electron. Agric..

[121]  Daming Xu,et al.  Near-infrared imaging to quantify the feeding behavior of fish in aquaculture , 2017, Comput. Electron. Agric..

[122]  Fan Liangzhong,et al.  Measuring feeding activity of fish in RAS using computer vision , 2014 .

[123]  Muhammad Imran Malik,et al.  Automatic fish species classification in underwater videos: exploiting pre-trained deep neural network models to compensate for limited labelled data , 2018 .

[124]  Xinting Yang,et al.  Feed intake prediction model for group fish using the MEA-BP neural network in intensive aquaculture , 2020 .

[125]  R. G. Randall,et al.  Stock assessment in inland fisheries: a foundation for sustainable use and conservation , 2016, Reviews in Fish Biology and Fisheries.

[126]  Chris Yakopcic,et al.  A State-of-the-Art Survey on Deep Learning Theory and Architectures , 2019, Electronics.

[127]  Jiunn-Ming Chen,et al.  Development of an adaptive neural-based fuzzy inference system for feeding decision-making assessment in silver perch (Bidyanus bidyanus) culture , 2015 .

[128]  Hang Li,et al.  Deep learning for natural language processing: advantages and challenges , 2018 .

[129]  James H Thrall,et al.  Artificial Intelligence and Machine Learning in Radiology: Opportunities, Challenges, Pitfalls, and Criteria for Success. , 2018, Journal of the American College of Radiology : JACR.

[130]  Prakash Choudhary,et al.  Image annotation: Then and now , 2018, Image Vis. Comput..

[131]  Sabeen Survery,et al.  Increased Permeability of the Aquaporin SoPIP2;1 by Mercury and Mutations in Loop A , 2016, Front. Plant Sci..

[132]  Jean Meunier,et al.  Learning cast shadow appearance for human posture recognition , 2017, Pattern Recognit. Lett..

[133]  Yan-Fu Kuo,et al.  Identifying the species of harvested tuna and billfish using deep convolutional neural networks , 2020 .

[134]  Yaoguang Wei,et al.  Research on a dissolved oxygen prediction method for recirculating aquaculture systems based on a convolution neural network , 2018, Comput. Electron. Agric..