A Review of Convolutional Neural Network Applied to Fruit Image Processing

Agriculture has always been an important economic and social sector for humans. Fruit production is especially essential, with a great demand from all households. Therefore, the use of innovative technologies is of vital importance for the agri-food sector. Currently artificial intelligence is one very important technological tool widely used in modern society. Particularly, Deep Learning (DL) has several applications due to its ability to learn robust representations from images. Convolutional Neural Networks (CNN) is the main DL architecture for image classification. Based on the great attention that CNNs have had in the last years, we present a review of the use of CNN applied to different automatic processing tasks of fruit images: classification, quality control, and detection. We observe that in the last two years (2019–2020), the use of CNN for fruit recognition has greatly increased obtaining excellent results, either by using new models or with pre-trained networks for transfer learning. It is worth noting that different types of images are used in datasets according to the task performed. Besides, this article presents the fundamentals, tools, and two examples of the use of CNNs for fruit sorting and quality control.

[1]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[2]  Donato Cascio,et al.  Deep Convolutional Neural Network for HEp-2 Fluorescence Intensity Classification , 2019, Applied Sciences.

[3]  Anuja Bhargava,et al.  Fruits and vegetables quality evaluation using computer vision: A review , 2021, J. King Saud Univ. Comput. Inf. Sci..

[4]  Richard K. G. Do,et al.  Convolutional neural networks: an overview and application in radiology , 2018, Insights into Imaging.

[5]  Xiangjun Zou,et al.  Guava Detection and Pose Estimation Using a Low-Cost RGB-D Sensor in the Field , 2019, Sensors.

[6]  Jun Zhou,et al.  Automatic apple recognition based on the fusion of color and 3D feature for robotic fruit picking , 2017, Comput. Electron. Agric..

[7]  Yudong Zhang,et al.  Fruit classification by biogeography‐based optimization and feedforward neural network , 2016, Expert Syst. J. Knowl. Eng..

[8]  Yudong Zhang,et al.  Fruit classification using computer vision and feedforward neural network , 2014 .

[9]  Michael I. Jordan,et al.  Machine learning: Trends, perspectives, and prospects , 2015, Science.

[10]  Menghan Hu,et al.  Optical non-destructive techniques for small berry fruits: A review , 2019, Artificial Intelligence in Agriculture.

[11]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[12]  Siddhartha S. Mehta,et al.  Deep Orange: Mask R-CNN based Orange Detection and Segmentation , 2019, IFAC-PapersOnLine.

[13]  Yurong Liu,et al.  A survey of deep neural network architectures and their applications , 2017, Neurocomputing.

[14]  Sidan Du,et al.  Image based fruit category classification by 13-layer deep convolutional neural network and data augmentation , 2019, Multimedia Tools and Applications.

[15]  K. Kheiralipour,et al.  Introducing new shape features for classification of cucumber fruit based on image processing technique and artificial neural networks , 2017 .

[16]  Yudong Zhang,et al.  Classification of Fruits Using Computer Vision and a Multiclass Support Vector Machine , 2012, Sensors.

[17]  Bruce MacDonald,et al.  Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms , 2019, Biosystems Engineering.

[18]  Raimund Leitner,et al.  Hyperspectral fruit and vegetable classification using convolutional neural networks , 2019, Comput. Electron. Agric..

[19]  Yang Yu,et al.  Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN , 2019, Comput. Electron. Agric..

[20]  Jong-Wook Kim,et al.  A Comparative Study of Deep CNN in Forecasting and Classifying the Macronutrient Deficiencies on Development of Tomato Plant , 2019, Applied Sciences.

[21]  Chunjiang Zhao,et al.  Intelligent alerting for fruit-melon lesion image based on momentum deep learning , 2015, Multimedia Tools and Applications.

[22]  Naoshi Kondo,et al.  Detecting greenhouse strawberries (mature and immature), using deep convolutional neural network , 2018, Engineering in Agriculture, Environment and Food.

[23]  Keng Siau,et al.  Artificial Intelligence, Machine Learning, Automation, Robotics, Future of Work and Future of Humanity: A Review and Research Agenda , 2019, J. Database Manag..

[24]  Jian Lian,et al.  Deep indicator for fine-grained classification of banana’s ripening stages , 2018, EURASIP Journal on Image and Video Processing.

[25]  Ting Liu,et al.  Recent advances in convolutional neural networks , 2015, Pattern Recognit..

[26]  Lip Yee Por,et al.  Investigation of Fusion Features for Apple Classification in Smart Manufacturing , 2019, Symmetry.

[27]  Ghulam Muhammad,et al.  Date Fruit Classification for Robotic Harvesting in a Natural Environment Using Deep Learning , 2019, IEEE Access.

[28]  Wenjuan Jia,et al.  An effective model based on Haar wavelet entropy and genetic algorithm for fruit identification , 2018 .

[29]  Gaoliang Peng,et al.  A deep convolutional neural network with new training methods for bearing fault diagnosis under noisy environment and different working load , 2018, Mechanical Systems and Signal Processing.

[30]  Rui Li,et al.  Improved Kiwifruit Detection Using Pre-Trained VGG16 With RGB and NIR Information Fusion , 2020, IEEE Access.

[31]  Yueju Xue,et al.  Detection of passion fruits and maturity classification using Red-Green-Blue Depth images , 2018, Biosystems Engineering.

[32]  James Patrick Underwood,et al.  Image Based Mango Fruit Detection, Localisation and Yield Estimation Using Multiple View Geometry , 2016, Sensors.

[33]  Arthur L. Samuel,et al.  Some studies in machine learning using the game of checkers , 2000, IBM J. Res. Dev..

[34]  Fernando Alfredo Auat Cheeín,et al.  Flexible system of multiple RGB-D sensors for measuring and classifying fruits in agri-food Industry , 2017, Comput. Electron. Agric..

[35]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[36]  Yi Chen,et al.  Fruit category classification via an eight-layer convolutional neural network with parametric rectified linear unit and dropout technique , 2018, Multimedia Tools and Applications.

[37]  S LewMichael,et al.  Deep learning for visual understanding , 2016 .

[38]  Katarzyna,et al.  A Vision-Based Method Utilizing Deep Convolutional Neural Networks for Fruit Variety Classification in Uncertainty Conditions of Retail Sales , 2019, Applied Sciences.

[39]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[40]  Mihai Oltean,et al.  Fruit recognition from images using deep learning , 2017, Acta Universitatis Sapientiae, Informatica.

[41]  Yudong Zhang,et al.  Fruit Classification by Wavelet-Entropy and Feedforward Neural Network Trained by Fitness-Scaled Chaotic ABC and Biogeography-Based Optimization , 2015, Entropy.

[42]  Maryam Rahnemoonfar,et al.  Deep Count: Fruit Counting Based on Deep Simulated Learning , 2017, Sensors.

[43]  Thiago T. Santos,et al.  Grape detection, segmentation and tracking using deep neural networks and three-dimensional association , 2019, Comput. Electron. Agric..

[44]  N RanjitK,et al.  Deep Features Based Approach for Fruit Disease Detection and Classification , 2019, International Journal of Computer Sciences and Engineering.

[45]  Jacques Wainer,et al.  Automatic fruit and vegetable classification from images , 2010 .

[46]  Bankim Patel,et al.  Machine Vision based Fruit Classification and Grading - A Review , 2017 .

[47]  Ang Wu,et al.  Detection of apple defect using laser-induced light backscattering imaging and convolutional neural network , 2020, Comput. Electr. Eng..

[48]  Qianhua He,et al.  Automatic Fruit Recognition Based on DCNN for Commercial Source Trace System , 2018, International Journal on Computational Science & Applications.

[49]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[50]  Matthias B. Hullin,et al.  Automated Phenotyping of Epicuticular Waxes of Grapevine Berries Using Light Separation and Convolutional Neural Networks , 2018, Comput. Electron. Agric..

[51]  P. From,et al.  Instance Segmentation and Localization of Strawberries in Farm Conditions for Automatic Fruit Harvesting , 2019, IFAC-PapersOnLine.

[52]  Vijay Kumar,et al.  Counting Apples and Oranges With Deep Learning: A Data-Driven Approach , 2017, IEEE Robotics and Automation Letters.

[53]  Douglas Chai,et al.  A comprehensive review of fruit and vegetable classification techniques , 2018, Image Vis. Comput..

[54]  Mihai Oltean.  Fruits 360 dataset , 2018 .

[55]  Christopher F. Lehnert,et al.  Towards Active Robotic Vision in Agriculture: A Deep Learning Approach to Visual Servoing in Occluded and Unstructured Protected Cropping Environments , 2019, IFAC-PapersOnLine.

[56]  Youngmin Park,et al.  Convolutional neural network based on an extreme learning machine for image classification , 2019, Neurocomputing.

[57]  Guangtao Zhai,et al.  Application of Deep Learning Architectures for Accurate and Rapid Detection of Internal Mechanical Damage of Blueberry Using Hyperspectral Transmittance Data , 2018, Sensors.

[58]  Mohammad Momeny,et al.  Classification of sour lemons based on apparent defects using stochastic pooling mechanism in deep convolutional neural networks , 2020 .

[59]  Michael S. Lew,et al.  Deep learning for visual understanding: A review , 2016, Neurocomputing.