Recognition of Bloom/Yield in Crop Images Using Deep Learning Models for Smart Agriculture: A Review

Precision agriculture is a crucial way to achieve greater yields by utilizing the natural deposits in a diverse environment. The yield of a crop may vary from year to year depending on the variations in climate, soil parameters and fertilizers used. Automation in the agricultural industry moderates the usage of resources and can increase the quality of food in the post-pandemic world. Agricultural robots have been developed for crop seeding, monitoring, weed control, pest management and harvesting. Physical counting of fruitlets, flowers or fruits at various phases of growth is labour intensive as well as an expensive procedure for crop yield estimation. Remote sensing technologies offer accuracy and reliability in crop yield prediction and estimation. The automation in image analysis with computer vision and deep learning models provides precise field and yield maps. In this review, it has been observed that the application of deep learning techniques has provided a better accuracy for smart farming. The crops taken for the study are fruits such as grapes, apples, citrus, tomatoes and vegetables such as sugarcane, corn, soybean, cucumber, maize, wheat. The research works which are carried out in this research paper are available as products for applications such as robot harvesting, weed detection and pest infestation. The methods which made use of conventional deep learning techniques have provided an average accuracy of 92.51%. This paper elucidates the diverse automation approaches for crop yield detection techniques with virtual analysis and classifier approaches. Technical hitches in the deep learning techniques have progressed with limitations and future investigations are also surveyed. This work highlights the machine vision and deep learning models which need to be explored for improving automated precision farming expressly during this pandemic.

[1]  Ginés García-Mateos,et al.  Comparison of Different Classifiers and the Majority Voting Rule for the Detection of Plum Fruits in Garden Conditions , 2019, Remote. Sens..

[2]  Pedro Dinis Gaspar,et al.  Crop Yield Estimation Using Deep Learning Based on Climate Big Data and Irrigation Scheduling , 2021, Energies.

[3]  He Li,et al.  Application of Crop Model Data Assimilation With a Particle Filter for Estimating Regional Winter Wheat Yields , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[4]  David C. Slaughter,et al.  Tractor-based Real-time Kinematic-Global Positioning System (RTK-GPS) guidance system for geospatial mapping of row crop transplant , 2012 .

[5]  Dejan Kaljaca,et al.  Coverage trajectory planning for a bush trimming robot arm , 2019, J. Field Robotics.

[6]  Hermano Igo Krebs,et al.  Robotics for Sugarcane Cultivation: Analysis of Billet Quality using Computer Vision , 2018, IEEE Robotics and Automation Letters.

[7]  Bini D,et al.  Machine Vision and Machine Learning for Intelligent Agrobots: A review , 2020, 2020 5th International Conference on Devices, Circuits and Systems (ICDCS).

[8]  Fumiya Iida,et al.  A field‐tested robotic harvesting system for iceberg lettuce , 2019, J. Field Robotics.

[9]  Scarlett Liu,et al.  Automatic grape bunch detection in vineyards with an SVM classifier , 2015, J. Appl. Log..

[10]  Fenghua Huang,et al.  Tomato Disease Detection and Classification by Deep Learning , 2020, 2020 International Conference on Big Data, Artificial Intelligence and Internet of Things Engineering (ICBAIE).

[11]  Grzegorz Cielniak,et al.  Analysis of Morphology-Based Features for Classification of Crop and Weeds in Precision Agriculture , 2018, IEEE Robotics and Automation Letters.

[12]  Roemi Fernández,et al.  Automatic Detection of Field-Grown Cucumbers for Robotic Harvesting , 2018, IEEE Access.

[13]  Hui Cheng,et al.  Decentralized Full Coverage of Unknown Areas by Multiple Robots With Limited Visibility Sensing , 2019, IEEE Robotics and Automation Letters.

[14]  G. Meyer,et al.  Verification of color vegetation indices for automated crop imaging applications , 2008 .

[15]  Tarmo Lipping,et al.  Crop yield prediction with deep convolutional neural networks , 2019, Comput. Electron. Agric..

[16]  Xuan Liu,et al.  Automatic Detection of Single Ripe Tomato on Plant Combining Faster R-CNN and Intuitionistic Fuzzy Set , 2019, IEEE Access.

[17]  Ryosuke Shibasaki,et al.  Estimating crop yields with deep learning and remotely sensed data , 2015, 2015 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[18]  Yong He,et al.  Citrus yield estimation based on images processed by an Android mobile phone , 2013 .

[19]  M. S. Moran,et al.  Opportunities and limitations for image-based remote sensing in precision crop management , 1997 .

[20]  Qi Wang,et al.  Automated Crop Yield Estimation for Apple Orchards , 2012, ISER.

[21]  Jiangang Yang,et al.  Recent Advances in Intelligent Automated Fruit Harvesting Robots , 2019, The Open Agriculture Journal.

[22]  Mohsen Azadbakht,et al.  Machine Learning Regression Techniques for the Silage Maize Yield Prediction Using Time-Series Images of Landsat 8 OLI , 2018, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[23]  Sanjiv Singh,et al.  Modeling and Calibrating Visual Yield Estimates in Vineyards , 2012, FSR.

[24]  Lorenzo Bruzzone,et al.  Spiking Neural Networks for Crop Yield Estimation Based on Spatiotemporal Analysis of Image Time Series , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[25]  A. Strahler,et al.  Monitoring vegetation phenology using MODIS , 2003 .

[26]  Changki Mo,et al.  Design, integration, and field evaluation of a robotic apple harvester , 2017, J. Field Robotics.

[27]  Sungchan Oh,et al.  Unmanned aerial system based tomato yield estimation using machine learning , 2019, Defense + Commercial Sensing.

[28]  Nidhi Goel,et al.  Fuzzy classification of pre-harvest tomatoes for ripeness estimation - An approach based on automatic rule learning using decision tree , 2015, Appl. Soft Comput..

[29]  Konstantinos P. Ferentinos,et al.  Deep learning models for plant disease detection and diagnosis , 2018, Comput. Electron. Agric..

[30]  Avital Bechar,et al.  Robotic Disease Detection in Greenhouses: Combined Detection of Powdery Mildew and Tomato Spotted Wilt Virus , 2016, IEEE Robotics and Automation Letters.

[31]  Hong Cheng,et al.  Early Yield Prediction Using Image Analysis of Apple Fruit and Tree Canopy Features with Neural Networks , 2017, J. Imaging.

[32]  Pål Johan From,et al.  A low‐cost and efficient autonomous row‐following robot for food production in polytunnels , 2019, J. Field Robotics.

[33]  Ho Seok Ahn,et al.  Improvements to and large‐scale evaluation of a robotic kiwifruit harvester , 2019, J. Field Robotics.

[34]  G. Timár,et al.  Crop yield estimation by satellite remote sensing , 2004 .

[35]  K. Walsh,et al.  Deep Learning for Mango (Mangifera indica) Panicle Stage Classification , 2020 .

[36]  Mihai Oltean,et al.  Fruit recognition from images using deep learning , 2017, Acta Universitatis Sapientiae, Informatica.

[37]  Vijay Kumar,et al.  Monocular Camera Based Fruit Counting and Mapping With Semantic Data Association , 2018, IEEE Robotics and Automation Letters.

[38]  Juan Frausto-Solís,et al.  Predictive ability of machine learning methods for massive crop yield prediction , 2014 .

[39]  Juan D. González-Teruel,et al.  Segmentation of Multiple Tree Leaves Pictures with Natural Backgrounds using Deep Learning for Image-Based Agriculture Applications , 2019, Applied Sciences.

[40]  Meonghun Lee,et al.  Artificial Intelligence Approach for Tomato Detection and Mass Estimation in Precision Agriculture , 2020, Sustainability.

[41]  K. Chand,et al.  Crop Insurance in India: A Review of Pradhan Mantri Fasal Bima Yojana (PMFBY) , 2020, FIIB Business Review.

[42]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[43]  Won Suk Lee,et al.  Immature green citrus fruit detection and counting based on fast normalized cross correlation (FNCC) using natural outdoor colour images , 2016, Precision Agriculture.

[44]  Yael Edan,et al.  Human‐robot collaborative site‐specific sprayer , 2017, J. Field Robotics.

[45]  Samy S. Abu-Naser,et al.  Type of Tomato Classification Using Deep Learning , 2020 .

[46]  Guan Gui,et al.  Deep Learning Based Improved Classification System for Designing Tomato Harvesting Robot , 2018, IEEE Access.

[47]  Charlie C. L. Wang,et al.  Plant Phenotyping by Deep-Learning-Based Planner for Multi-Robots , 2019, IEEE Robotics and Automation Letters.

[48]  Tristan Perez,et al.  Fruit Quantity and Ripeness Estimation Using a Robotic Vision System , 2018, IEEE Robotics and Automation Letters.

[49]  Michihisa Iida,et al.  Rice Autonomous Harvesting: Operation Framework , 2017, J. Field Robotics.

[50]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[51]  D. Bochtis,et al.  Yield prediction in apple orchards based on image processing , 2011, Precision Agriculture.

[52]  Jun Sun,et al.  Detection of Key Organs in Tomato Based on Deep Migration Learning in a Complex Background , 2018, Agriculture.

[53]  Artzai Picón,et al.  Deep convolutional neural networks for mobile capture device-based crop disease classification in the wild , 2019, Comput. Electron. Agric..

[54]  Thiago T. Santos,et al.  Grape detection, segmentation and tracking using deep neural networks and three-dimensional association , 2019, Comput. Electron. Agric..

[55]  Nitaigour P. Mahalik,et al.  Autonomous Greenhouse Mobile Robot Driving Strategies From System Integration Perspective: Review and Application , 2015, IEEE/ASME Transactions on Mechatronics.

[56]  Q. Zhang,et al.  Sensors and systems for fruit detection and localization: A review , 2015, Comput. Electron. Agric..

[57]  Asher Bender,et al.  A high‐resolution, multimodal data set for agricultural robotics: A Ladybird's‐eye view of Brassica , 2019, J. Field Robotics.

[58]  Jun Liu,et al.  Tomato Diseases and Pests Detection Based on Improved Yolo V3 Convolutional Neural Network , 2020, Frontiers in Plant Science.

[59]  Shuyi Mao,et al.  A Mature-Tomato Detection Algorithm Using Machine Learning and Color Analysis † , 2019, Sensors.

[60]  Victor Alchanatis,et al.  Image fusion of visible and thermal images for fruit detection. , 2009 .

[61]  D. Bulanon,et al.  A Segmentation Algorithm for the Automatic Recognition of Fuji Apples at Harvest , 2002 .

[62]  C. Glasbey,et al.  Automatic fruit recognition and counting from multiple images , 2014 .

[63]  Satya Prakash Yadav,et al.  Estimation of the chlorophyll content of micropropagated potato plants using RGB based image analysis , 2010, Plant Cell, Tissue and Organ Culture (PCTOC).

[64]  Deepak Murugan,et al.  Development of an Adaptive Approach for Precision Agriculture Monitoring with Drone and Satellite Data , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[65]  P. Atkinson,et al.  Remote sensing of mangrove forest phenology and its environmental drivers , 2018 .

[66]  Yael Edan,et al.  Harvesting Robots for High‐value Crops: State‐of‐the‐art Review and Challenges Ahead , 2014, J. Field Robotics.

[67]  Sanjiv Singh,et al.  Autonomous Orchard Vehicles for Specialty Crops Production , 2011 .

[68]  J. Senthilnath,et al.  Detection of tomatoes using spectral-spatial methods in remotely sensed RGB images captured by UAV , 2016 .

[69]  Henry Medeiros,et al.  Multispecies Fruit Flower Detection Using a Refined Semantic Segmentation Network , 2018, IEEE Robotics and Automation Letters.

[70]  Jochen Hemming,et al.  Fruit Detectability Analysis for Different Camera Positions in Sweet-Pepper † , 2014, Sensors.

[71]  Benjamin Fernandez,et al.  A Simplified Optimal Path Following Controller for an Agricultural Skid-Steering Robot , 2019, IEEE Access.

[72]  Denis Stajnko,et al.  Original papers: Detecting fruits in natural scenes by using spatial-frequency based texture analysis and multiview geometry , 2011 .

[73]  Xiangjun Zou,et al.  Fruit detection in natural environment using partial shape matching and probabilistic Hough transform , 2019, Precision Agriculture.

[74]  Jun Zhou,et al.  Automatic Recognition of Ripening Tomatoes by Combining Multi-Feature Fusion with a Bi-Layer Classification Strategy for Harvesting Robots , 2019, Sensors.

[75]  Won Suk Lee,et al.  Green citrus detection using 'eigenfruit', color and circular Gabor texture features under natural outdoor conditions , 2011 .

[76]  Yibin Ying,et al.  Research on image segmentation methods of tomato in natural conditions , 2011, 2011 4th International Congress on Image and Signal Processing.

[77]  Jong-Wook Kim,et al.  A Comparative Study of Deep CNN in Forecasting and Classifying the Macronutrient Deficiencies on Development of Tomato Plant , 2019, Applied Sciences.

[78]  Giovanni Laneve,et al.  Agricultural Monitoring, an Automatic Procedure for Crop Mapping and Yield Estimation: The Great Rift Valley of Kenya Case , 2019, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[79]  Viacheslav I. Adamchuk,et al.  On-the-go soil sensors for precision agriculture , 2004 .

[80]  Reza Ehsani,et al.  A methodology for fresh tomato maturity detection using computer vision , 2018, Comput. Electron. Agric..

[81]  Gonzalo Pajares,et al.  Machine-Vision Systems Selection for Agricultural Vehicles: A Guide , 2016, J. Imaging.

[82]  Jordi Llorens,et al.  Fruit detection in an apple orchard using a mobile terrestrial laser scanner , 2019, Biosystems Engineering.

[83]  Nari Kim,et al.  Machine Learning Approaches to Corn Yield Estimation Using Satellite Images and Climate Data :A Case of Iowa State , 2016 .

[84]  Samuel Williams,et al.  A Robot System for Pruning Grape Vines , 2017, J. Field Robotics.

[85]  Wolfram Burgard,et al.  Crop Row Detection on Tiny Plants With the Pattern Hough Transform , 2018, IEEE Robotics and Automation Letters.

[86]  Seishi Ninomiya,et al.  On Plant Detection of Intact Tomato Fruits Using Image Analysis and Machine Learning Methods , 2014, Sensors.

[87]  Qi Jing,et al.  Crop Yield Estimation Using Time-Series MODIS Data and the Effects of Cropland Masks in Ontario, Canada , 2019, Remote. Sens..

[88]  Liang Gong,et al.  Detecting tomatoes in greenhouse scenes by combining AdaBoost classifier and colour analysis , 2016 .

[89]  Tristan Perez,et al.  Efficacy of Mechanical Weeding Tools: A Study Into Alternative Weed Management Strategies Enabled by Robotics , 2018, IEEE Robotics and Automation Letters.

[90]  Hui Huang,et al.  An Autonomous Fruit and Vegetable Harvester with a Low-Cost Gripper Using a 3D Sensor , 2019, Sensors.

[91]  C. Walthall,et al.  Artificial neural networks for corn and soybean yield prediction , 2005 .

[92]  R. Bhargavi,et al.  A novel approach for efficient crop yield prediction , 2019, Comput. Electron. Agric..

[93]  Thanasis Hadzilacos,et al.  Design and development of a semi‐autonomous agricultural vineyard sprayer: Human–robot interaction aspects , 2017, J. Field Robotics.

[94]  Wiqas Ghai,et al.  Performance analysis of deep learning CNN models for disease detection in plants using image segmentation , 2020, Information Processing in Agriculture.

[95]  Gwo-Jiun Horng,et al.  The Smart Image Recognition Mechanism for Crop Harvesting System in Intelligent Agriculture , 2020, IEEE Sensors Journal.

[96]  Jesus Soria-Ruiz,et al.  Maize crop yield estimation with remote sensing and empirical models , 2017, 2017 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[97]  Abdelouahab Moussaoui,et al.  Deep Learning for Tomato Diseases: Classification and Symptoms Visualization , 2017, Appl. Artif. Intell..

[98]  Jinhai Li,et al.  Review of Wheeled Mobile Robots’ Navigation Problems and Application Prospects in Agriculture , 2018, IEEE Access.

[99]  Maryam Rahnemoonfar,et al.  Deep Count: Fruit Counting Based on Deep Simulated Learning , 2017, Sensors.

[100]  Malrey Lee,et al.  An yield estimation in citrus orchards via fruit detection and counting using image processing , 2017, Comput. Electron. Agric..

[101]  Dean Zhao,et al.  Grasping damage analysis of apple by end-effector in harvesting robot , 2017 .

[102]  Arthur Zanatta Da Costa,et al.  Computer vision based detection of external defects on tomatoes using deep learning , 2020 .

[103]  Yael Edan,et al.  Development of a sweet pepper harvesting robot , 2020, J. Field Robotics.

[104]  Eduard Clotet,et al.  Vineyard Yield Estimation Based on the Analysis of High Resolution Images Obtained with Artificial Illumination at Night , 2015, Sensors.

[105]  Raj S. Chhikara,et al.  Use of satellite spectral data in crop yield estimation surveys , 1992 .

[106]  Hyoung Il Son,et al.  Unmanned Aerial Vehicles in Agriculture: A Review of Perspective of Platform, Control, and Applications , 2019, IEEE Access.

[107]  Yael Edan,et al.  Changing Task Objectives for Improved Sweet Pepper Detection for Robotic Harvesting , 2016, IEEE Robotics and Automation Letters.

[108]  Fengguo Li,et al.  Leaf Segmentation and Classification with a Complicated Background Using Deep Learning , 2020, Agronomy.

[109]  Lars Grimstad,et al.  An autonomous strawberry‐harvesting robot: Design, development, integration, and field evaluation , 2019, J. Field Robotics.

[110]  R. Zhou,et al.  Using colour features of cv. ‘Gala’ apple fruits in an orchard in image processing to predict yield , 2012, Precision Agriculture.

[111]  Dennis Jarvis,et al.  Estimation of mango crop yield using image analysis - Segmentation method , 2013 .

[112]  Shima Ramesh Maniyath,et al.  Plant Disease Detection Using Machine Learning , 2018, 2018 International Conference on Design Innovations for 3Cs Compute Communicate Control (ICDI3C).

[113]  Miguel Torres-Torriti,et al.  Agricultural service unit motion planning under harvesting scheduling and terrain constraints , 2017, J. Field Robotics.

[114]  Young K. Chang,et al.  Current and future applications of statistical machine learning algorithms for agricultural machine vision systems , 2019, Comput. Electron. Agric..