Classification and detection of insects from field images using deep learning for smart pest management: A systematic review

Abstract Insect pest is one of the main causes affecting agricultural crop yield and quality all over the world. Rapid and reliable insect pest monitoring plays a crucial role in population prediction and control actions. The great breakthrough of deep learning (DL) technology has resulted in its successful applications in various fields, including automatic insect pest monitoring. DL creates both new strengths and a series of challenges for data processing in smart pest monitoring (SPM). This review outlines the technical methods of DL frameworks and their applications in SPM with emphasis on insect pest classification and detection using field images. The methodologies and technical details evolved in insect pest classification and detection using DL are summarized and distilled during different processing stages: image acquisition, data preprocessing and modeling techniques. Finally, a general framework is provided to facilitate the smart insect monitoring and future challenges and trends are highlighted. In a word, our purpose is to provide researchers and technicians with a better understanding of DL techniques and their state-of-art achievements in SPM, which can promote the implement of various SPM applications.

[1]  Y. A. Nanehkaran,et al.  Crop pest recognition using attention-embedded lightweight network under field conditions , 2021, Applied Entomology and Zoology.

[2]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[3]  F. Verheggen,et al.  Insect pest monitoring with camera-equipped traps: strengths and limitations , 2020, Journal of Pest Science.

[4]  Xiaoguang Zhou,et al.  Detection and Identification of Stored-Grain Insects Using Deep Learning: A More Effective Neural Network , 2020, IEEE Access.

[5]  Che-Lun Hung,et al.  A Smartphone-Based Application for Scale Pest Detection Using Multiple-Object Detection Methods , 2021, Electronics.

[6]  Bashir Al-Diri,et al.  Mobile Real-Time Grasshopper Detection and Data Aggregation Framework , 2020, Scientific Reports.

[7]  Bogdan Raducanu,et al.  Saliency for Fine-grained Object Recognition in Domains with Scarce Training Data , 2018, Pattern Recognit..

[8]  Yu Jian,et al.  A Crop Pests Image Classification Algorithm Based on Deep Convolutional Neural Network , 2017 .

[9]  Hemerson Pistori,et al.  Detection and classification of soybean pests using deep learning with UAV images , 2020, Comput. Electron. Agric..

[10]  Thenmozhi Kasinathan,et al.  Insect classification and detection in field crops using modern machine learning techniques , 2020 .

[11]  Aharon Bar-Hillel,et al.  Object detection in agricultural contexts: A multiple resolution benchmark and comparison to human , 2021, Comput. Electron. Agric..

[12]  Ayad Almryad,et al.  Automatic identification for field butterflies by convolutional neural networks , 2020 .

[13]  Bing Wang,et al.  Automatic Localization and Count of Agricultural Crop Pests Based on an Improved Deep Learning Pipeline , 2019, Scientific Reports.

[14]  Rui Li,et al.  Pest24: A large-scale very small object data set of agricultural pests for multi-target detection , 2020, Comput. Electron. Agric..

[15]  Loreto Susperregi,et al.  A Benchmarking of Learning Strategies for Pest Detection and Identification on Tomato Plants for Autonomous Scouting Robots Using Internal Databases , 2019, J. Sensors.

[16]  Arief Setyanto,et al.  Data augmentation for automated pest classification in Mango farms , 2020, Comput. Electron. Agric..

[17]  Fahad Shahbaz Khan,et al.  PSC-Net: Learning Part Spatial Co-occurence for Occluded Pedestrian Detection , 2020, ArXiv.

[18]  Patrick Mäder,et al.  Machine learning for image based species identification , 2018, Methods in Ecology and Evolution.

[19]  Jayme Garcia Arnal Barbedo,et al.  Influence of image quality on the identification of psyllids using convolutional neural networks , 2019, Biosystems Engineering.

[20]  Ghiseok Kim,et al.  Automatic Pest Counting from Pheromone Trap Images Using Deep Learning Object Detectors for Matsucoccus thunbergianae Monitoring , 2021, Insects.

[21]  Youhua Zhang,et al.  Agricultural Pest Super-Resolution and Identification With Attention Enhanced Residual and Dense Fusion Generative and Adversarial Network , 2020, IEEE Access.

[22]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[23]  Haiyang Zhou,et al.  A smart-vision algorithm for counting whiteflies and thrips on sticky traps using two-dimensional Fourier transform spectrum , 2017 .

[24]  Rutu Gandhi,et al.  Plant disease detection using CNNs and GANs as an augmentative approach , 2018, 2018 IEEE International Conference on Innovative Research and Development (ICIRD).

[25]  Xinting Yang,et al.  Small-sample learning with salient-region detection and center neighbor loss for insect recognition in real-world complex scenarios , 2021, Comput. Electron. Agric..

[26]  Ezzatollah Askari Asli-Ardeh,et al.  Citrus pests classification using an ensemble of deep learning models , 2021, Comput. Electron. Agric..

[27]  Ming-Ming Cheng,et al.  IP102: A Large-Scale Benchmark Dataset for Insect Pest Recognition , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[28]  Rui Li,et al.  Multi-level learning features for automatic classification of field crop pests , 2018, Comput. Electron. Agric..

[29]  Ta-Te Lin,et al.  Online semi-supervised learning applied to an automated insect pest monitoring system , 2021 .

[30]  Thomas G. Dietterich,et al.  Segmentation of touching insects based on optical flow and NCuts , 2013 .

[31]  Taghi M. Khoshgoftaar,et al.  A survey on Image Data Augmentation for Deep Learning , 2019, Journal of Big Data.

[32]  Saeid Minaei,et al.  Vision-based pest detection based on SVM classification method , 2017, Comput. Electron. Agric..

[33]  Deng Limiao,et al.  Recognition pest by image-based transfer learning. , 2019, Journal of the science of food and agriculture.

[34]  Jun Liu,et al.  Tomato Diseases and Pests Detection Based on Improved Yolo V3 Convolutional Neural Network , 2020, Frontiers in Plant Science.

[35]  Tae-Soo Chon,et al.  Automatic identification and counting of small size pests in greenhouse conditions with low computational cost , 2015, Ecol. Informatics.

[36]  Zhongzhi Han,et al.  Research on insect pest image detection and recognition based on bio-inspired methods , 2018 .

[37]  A. Maki,et al.  Automated Taxonomic Identification of Insects with Expert-Level Accuracy Using Effective Feature Transfer from Convolutional Networks , 2019, Systematic biology.

[38]  M. P. Gopinath,et al.  RETRACTED ARTICLE: Classification of pest detection in paddy crop based on transfer learning approach , 2021, Acta Agriculturae Scandinavica, Section B — Soil & Plant Science.

[39]  U. Srinivasulu Reddy,et al.  Crop pest classification based on deep convolutional neural network and transfer learning , 2019, Comput. Electron. Agric..

[40]  Rafael Rieder,et al.  Computer vision and artificial intelligence in precision agriculture for grain crops: A systematic review , 2018, Comput. Electron. Agric..

[41]  Vimal K. Shrivastava,et al.  Transfer Learning-Based Framework for Classification of Pest in Tomato Plants , 2020, Appl. Artif. Intell..

[42]  Jiangning Wang,et al.  A new automatic identification system of insect images at the order level , 2012, Knowl. Based Syst..

[43]  Greg Bishop-Hurley,et al.  Dynamic cattle behavioural classification using supervised ensemble classifiers , 2015, Comput. Electron. Agric..

[44]  Stavros Ntalampiras.  One-shot learning for acoustic diagnosis of industrial machines , 2021, Expert Syst. Appl..

[45]  Chen Ning,et al.  Survey of pedestrian detection with occlusion , 2020, Complex & Intelligent Systems.

[46]  Kazuhiko Kawamoto,et al.  Generative and self-supervised domain adaptation for one-stage object detection , 2021, Array.

[47]  Yu Sun,et al.  Automatic in-trap pest detection using deep learning for pheromone-based Dendroctonus valens monitoring , 2018, Biosystems Engineering.

[48]  Jiqing Chen,et al.  Detecting ripe fruits under natural occlusion and illumination conditions , 2021, Computers and Electronics in Agriculture.

[49]  Bo Chen,et al.  MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications , 2017, ArXiv.

[50]  Qingxuan Jia,et al.  Multi-scale detection of stored-grain insects for intelligent monitoring , 2020, Comput. Electron. Agric..

[51]  Marely Lee,et al.  Citrus Pests and Diseases Recognition Model Using Weakly Dense Connected Convolution Network , 2019, Sensors.

[52]  Johannes Fahrentrapp,et al.  Deep learning for automated detection of Drosophila suzukii: potential for UAV‐based monitoring , 2020, Pest management science.

[53]  Jun Liu,et al.  Plant diseases and pests detection based on deep learning: a review , 2021, Plant Methods.

[54]  Yiannis Ampatzidis,et al.  Automated vision-based system for monitoring Asian citrus psyllid in orchards utilizing artificial intelligence , 2019, Comput. Electron. Agric..

[55]  Xinting Yang,et al.  Using an Improved YOLOv4 Deep Learning Network for Accurate Detection of Whitefly and Thrips on Sticky Trap Images , 2021 .

[56]  Zhiyan Zhou,et al.  Brown rice planthopper (Nilaparvata lugens Stal) detection based on deep learning , 2020, Precision Agriculture.

[57]  Zhongfei Zhang,et al.  Improved prototypical networks for few-Shot learning , 2020, Pattern Recognit. Lett..

[58]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[59]  K. Ganeshaiah,et al.  Identification of Indian Butterflies and Moths with Deep Convolutional Neural Networks x , 2020, Current Science.

[60]  Chengjun Xie,et al.  AF-RCNN: An anchor-free convolutional neural network for multi-categories agricultural pest detection , 2020, Comput. Electron. Agric..

[61]  Hyeonjoon Moon,et al.  Crop pest recognition in natural scenes using convolutional neural networks , 2020, Comput. Electron. Agric..

[62]  Jing Li,et al.  Seasonal monitoring of Drosophila suzukii and its non‐crop hosts in Wuhu, Eastern China , 2020 .

[63]  Loris Nanni,et al.  Research on insect pest image detection and recognition based on bio-inspired methods , 2019, Ecol. Informatics.

[64]  Fangyuan Wang,et al.  PestNet: An End-to-End Deep Learning Approach for Large-Scale Multi-Class Pest Detection and Classification , 2019, IEEE Access.

[65]  Sreekala G. Bajwa,et al.  Detection of soybean aphids in a greenhouse using an image processing technique , 2017, Comput. Electron. Agric..

[66]  Mohamed Elhoseny,et al.  Real-time detection of cole diseases and insect pests in wireless sensor networks , 2019, J. Intell. Fuzzy Syst..

[67]  Seong Joon Yoo,et al.  Transfer Learning-Based Search Model for Hot Pepper Diseases and Pests , 2020 .

[68]  Rujing Wang,et al.  S-RPN: Sampling-balanced region proposal network for small crop pest detection , 2021, Comput. Electron. Agric..

[69]  Yueju Xue,et al.  Center clustering network improves piglet counting under occlusion , 2021, Comput. Electron. Agric..

[70]  Chuan-Yu Chang,et al.  Identification of Fruit Tree Pests With Deep Learning on Embedded Drone to Achieve Accurate Pesticide Spraying , 2021, IEEE Access.

[71]  Li Rui,et al.  A coarse-to-fine network for aphid recognition and detection in the field , 2019, Biosystems Engineering.

[72]  Azree Nazri,et al.  PENYEK: Automated brown planthopper detection from imperfect sticky pad images using deep convolutional neural network , 2018, PloS one.

[73]  Alejandro López,et al.  Combination of image processing and artificial neural networks as a novel approach for the identification of Bemisia tabaci and Frankliniella occidentalis on sticky traps in greenhouse agriculture , 2016, Comput. Electron. Agric..

[74]  Rafael Rieder,et al.  Automatic identification of insects from digital images: A survey , 2020, Comput. Electron. Agric..

[75]  Alexandros Iosifidis,et al.  Deep learning and computer vision will transform entomology , 2020, Proceedings of the National Academy of Sciences.

[76]  Oriol Vinyals,et al.  Matching Networks for One Shot Learning , 2016, NIPS.

[77]  Chuan-Yu Chang,et al.  An AIoT Based Smart Agricultural System for Pests Detection , 2020, IEEE Access.

[78]  Xi Cheng,et al.  Pest identification via deep residual learning in complex background , 2017, Comput. Electron. Agric..

[79]  Díbio Leandro Borges,et al.  Cotton pests classification in field-based images using deep residual networks , 2020, Comput. Electron. Agric..

[80]  Yangyang Fan,et al.  Application of Deep Learning in Integrated Pest Management: A Real-Time System for Detection and Diagnosis of Oilseed Rape Pests , 2019, Mob. Inf. Syst..

[81]  Malrey Lee,et al.  Classification Accuracy Improvement for Small-Size Citrus Pests and Diseases Using Bridge Connections in Deep Neural Networks , 2020, Sensors.

[82]  Chenglu Wen,et al.  Image-based orchard insect automated identification and classification method , 2012 .

[83]  Jin Wang,et al.  Common pests image recognition based on deep convolutional neural network , 2020, Comput. Electron. Agric..

[84]  Rui Li,et al.  AgriPest: A Large-Scale Domain-Specific Benchmark Dataset for Practical Agricultural Pest Detection in the Wild , 2021, Sensors.

[85]  Huajian Liu,et al.  Proximal detecting invertebrate pests on crops using a deep residual convolutional neural network trained by virtual images , 2021 .

[86]  Siow-Wee Chang,et al.  Integrated Analysis of Machine Learning and Deep Learning in Chili Pest and Disease Identification. , 2020, Journal of the science of food and agriculture.

[87]  Fangyuan Wang,et al.  An Effective Data Augmentation Strategy for CNN-Based Pest Localization and Recognition in the Field , 2019, IEEE Access.

[88]  Huan Zhang,et al.  Localization and Classification of Paddy Field Pests using a Saliency Map and Deep Convolutional Neural Network , 2016, Scientific Reports.

[89]  Romain Raveaux,et al.  A survey on image-based insect classification , 2017, Pattern Recognit..

[90]  Fatih Varcin,et al.  Crop pest classification with a genetic algorithm-based weighted ensemble of deep convolutional neural networks , 2020, Comput. Electron. Agric..

[91]  Mohamed Esmail Karar,et al.  A new mobile application of agricultural pests recognition using deep learning in cloud computing system , 2021 .

[92]  Graham W. Taylor,et al.  Automatic moth detection from trap images for pest management , 2016, Comput. Electron. Agric..

[93]  Xinting Yang,et al.  Field detection of tiny pests from sticky trap images using deep learning in agricultural greenhouse , 2021, Comput. Electron. Agric..

[94]  Joan Bruna,et al.  Few-Shot Learning with Graph Neural Networks , 2017, ICLR.

[95]  Ta-Te Lin,et al.  Automatic greenhouse insect pest detection and recognition based on a cascaded deep learning classification method , 2020 .

[96]  Fangyuan Wang,et al.  A Recognition Method for Rice Plant Diseases and Pests Video Detection Based on Deep Convolutional Neural Network , 2020, Sensors.

[97]  Po Yang,et al.  Fusing multi-scale context-aware information representation for automatic in-field pest detection and recognition , 2020, Comput. Electron. Agric..

[98]  Yang Li,et al.  Few-shot cotton pest recognition and terminal realization , 2020, Comput. Electron. Agric..

[99]  Jie You,et al.  Offline mobile diagnosis system for citrus pests and diseases using deep compression neural network , 2020, IET Comput. Vis..

[100]  Jinyan Li,et al.  Automatic classification for field crop insects via multiple-task sparse representation and multiple-kernel learning , 2015, Comput. Electron. Agric..

[101]  Sang Cheol Kim,et al.  A Robust Deep-Learning-Based Detector for Real-Time Tomato Plant Diseases and Pests Recognition , 2017, Sensors.

[102]  Mohammed Eunus Ali,et al.  Identification and recognition of rice diseases and pests using convolutional neural networks , 2018, Biosystems Engineering.

[103]  J. Liu,et al.  Early real-time detection algorithm of tomato diseases and pests in the natural environment , 2021, Plant methods.

[104]  Hung-Yu Tseng,et al.  Cross-Domain Few-Shot Classification via Learned Feature-Wise Transformation , 2020, ICLR.