Fast and accurate detection of kiwifruit in orchard using improved YOLOv3-tiny model

Automatic detection of kiwifruit in the orchard is challenging because illumination varies through the day and night and because of color similarity between kiwifruit and the complex background of leaves, branches and stems. Also, kiwifruits grow in clusters, which may result in having occluded and touching fruits. A fast and accurate object detection algorithm was developed to automatically detect kiwifruits in the orchard by improving the YOLOv3-tiny model. Based on the characteristics of kiwifruit images, two convolutional kernels of 3 × 3 and 1 × 1 were added to the fifth and sixth convolution layers of the YOLOv3-tiny model, respectively, to develop a deep YOLOv3-tiny (DY3TNet) model. It takes multiple 1 × 1 convolutional layers in intermediate layers of the network to reduce the computational complexity. Testing images captured from day and night and comparing with other deep learning models, namely, Faster R-CNN with ZFNet, Faster R-CNN with VGG16, YOLOv2 and YOLOv3-tiny, the DY3TNet model achieved the highest average precision of 0.9005 with the smallest data weight of 27 MB. Furthermore, it took only 34 ms on average to process an image of a resolution of 2352 × 1568 pixels. The DY3TNet model, along with the YOLOv3-tiny model, showed better performance on images captured with flash than those without. Moreover, the experiments indicated that the image augmentation process could improve the detection performance, and a simple lighting arrangement could improve the success rate of detection in the orchard. The experimental results demonstrated that the improved DY3TNet model is small and efficient and that it would increase the applicability of real-time kiwifruit detection in the orchard even when small hardware devices are used.

[1]  Edwin Valarezo,et al.  Simultaneous Detection and Classification of Breast Masses in Digital Mammograms via a Deep Learning YOLO-based CAD System , 2018, Comput. Methods Programs Biomed..

[2]  Fernando Auat Cheein,et al.  Comparison of convolutional neural networks in fruit detection and counting: A comprehensive evaluation , 2020, Comput. Electron. Agric..

[3]  John T. Wen,et al.  Motion Blur-Based State Estimation , 2016, IEEE Transactions on Control Systems Technology.

[4]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Wang Bin,et al.  Kiwifruit recognition at nighttime using artificial lighting based on machine vision. , 2015 .

[6]  Ashwin Kothari,et al.  YOLO based Human Action Recognition and Localization , 2018 .

[7]  Yong He,et al.  Detection of orchard citrus fruits using a monocular machine vision-based method for automatic fruit picking applications , 2018, Comput. Electron. Agric..

[8]  Michael S. Bernstein,et al.  ImageNet Large Scale Visual Recognition Challenge , 2014, International Journal of Computer Vision.

[9]  Ho Seok Ahn,et al.  Improvements to and large‐scale evaluation of a robotic kiwifruit harvester , 2019, J. Field Robotics.

[10]  Yuyu Tian,et al.  Detection and segmentation of overlapped fruits based on optimized mask R-CNN application in apple harvesting robot , 2020, Comput. Electron. Agric..

[11]  Chen Sun,et al.  Revisiting Unreasonable Effectiveness of Data in Deep Learning Era , 2017, 2017 IEEE International Conference on Computer Vision (ICCV).

[12]  Christina Thorpe,et al.  HaRD: a heterogeneity-aware replica deletion for HDFS , 2019, Journal of Big Data.

[13]  Eric Fleury,et al.  MultiAspect Graphs: Algebraic representation and algorithms , 2015, Algorithms.

[14]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Changki Mo,et al.  Design, integration, and field evaluation of a robotic apple harvester , 2017, J. Field Robotics.

[16]  D. Annane,et al.  Nocturnal mechanical ventilation for chronic hypoventilation in patients with neuromuscular and chest wall disorders. , 2014, The Cochrane database of systematic reviews.

[17]  John D. Austin,et al.  Adaptive histogram equalization and its variations , 1987 .

[18]  Geoff S. Nitschke,et al.  Improving Deep Learning with Generic Data Augmentation , 2018, 2018 IEEE Symposium Series on Computational Intelligence (SSCI).

[19]  J. Bellarby,et al.  Land-use changes from arable crop to kiwi-orchard increased nutrient surpluses and accumulation in soils , 2016 .

[20]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[21]  Junxing Zhang,et al.  Fast vehicle logo detection in complex scenes , 2019, Optics & Laser Technology.

[22]  Xiangjun Zou,et al.  Color-, depth-, and shape-based 3D fruit detection , 2019, Precision Agriculture.

[23]  Guoxu Liu,et al.  YOLO-Tomato: A Robust Algorithm for Tomato Detection Based on YOLOv3 , 2020, Sensors.

[24]  Wei Liu,et al.  SSD: Single Shot MultiBox Detector , 2015, ECCV.

[25]  Jianming Zhang,et al.  A Real-Time Chinese Traffic Sign Detection Algorithm Based on Modified YOLOv2 , 2017, Algorithms.

[26]  Alvy Ray Smith,et al.  Color gamut transform pairs , 1978, SIGGRAPH.

[27]  A R Smith,et al.  Color Gamut Transformation Pairs , 1978 .

[28]  Xin Zhang,et al.  Multi-class fruit-on-plant detection for apple in SNAP system using Faster R-CNN , 2020, Comput. Electron. Agric..

[29]  Won Suk Lee,et al.  Detection and counting of immature green citrus fruit based on the Local Binary Patterns (LBP) feature using illumination-normalized images , 2018, Precision Agriculture.

[30]  Ali Farhadi,et al.  YOLO9000: Better, Faster, Stronger , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Y. Gejima,et al.  Mechanized technologies for scaffolding cultivation in the kiwifruit industry: A review , 2018, Information Processing in Agriculture.

[32]  Xiaoyang Liu,et al.  The recognition of apple fruits in plastic bags based on block classification , 2017, Precision Agriculture.

[33]  Toby P. Breckon,et al.  Using Deep Convolutional Neural Network Architectures for Object Classification and Detection Within X-Ray Baggage Security Imagery , 2018, IEEE Transactions on Information Forensics and Security.

[34]  Andrew Zisserman,et al.  Two-Stream Convolutional Networks for Action Recognition in Videos , 2014, NIPS.

[35]  Josep Ramon Morros,et al.  Fruit detection and 3D location using instance segmentation neural networks and structure-from-motion photogrammetry , 2020, Comput. Electron. Agric..

[36]  Jeremiah Neubert,et al.  Automatic diagnosis of melanoma from dermoscopic image using real-time object detection , 2018, 2018 52nd Annual Conference on Information Sciences and Systems (CISS).

[37]  Bernt Schiele,et al.  CityPersons: A Diverse Dataset for Pedestrian Detection , 2017, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[38]  Rui Li,et al.  Classification of Kiwifruit Grades Based on Fruit Shape Using a Single Camera , 2016, Sensors.

[39]  Ahmad Al-Mallahi,et al.  A novel image processing algorithm to separate linearly clustered kiwifruits , 2019, Biosystems Engineering.

[40]  En Li,et al.  Apple detection during different growth stages in orchards using the improved YOLO-V3 model , 2019, Comput. Electron. Agric..

[41]  Xin Zhang,et al.  Faster R–CNN–based apple detection in dense-foliage fruiting-wall trees using RGB and depth features for robotic harvesting , 2020 .

[42]  Zhihao Liu,et al.  Kiwifruit yield estimation using image processing by an Android mobile phone , 2018 .

[43]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[44]  Tristan Perez,et al.  DeepFruits: A Fruit Detection System Using Deep Neural Networks , 2016, Sensors.

[45]  R. Linker.  Machine learning based analysis of night-time images for yield prediction in apple orchard , 2018 .

[46]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[47]  Kamel Barkaoui,et al.  Performability evaluation of server virtualized systems under bursty workload , 2018 .

[48]  Taghi M. Khoshgoftaar,et al.  A survey on Image Data Augmentation for Deep Learning , 2019, Journal of Big Data.

[49]  K. Walsh,et al.  Deep learning for real-time fruit detection and orchard fruit load estimation: benchmarking of ‘MangoYOLO’ , 2019, Precision Agriculture.

[50]  James Patrick Underwood,et al.  Image Segmentation for Fruit Detection and Yield Estimation in Apple Orchards , 2016, J. Field Robotics.

[51]  Rui Li,et al.  Kiwifruit detection in field images using Faster R-CNN with VGG16 , 2019, IFAC-PapersOnLine.

[52]  Rachel Huang,et al.  YOLO-LITE: A Real-Time Object Detection Algorithm Optimized for Non-GPU Computers , 2018, 2018 IEEE International Conference on Big Data (Big Data).

[53]  Yang Yu,et al.  Fruit detection for strawberry harvesting robot in non-structural environment based on Mask-RCNN , 2019, Comput. Electron. Agric..

[54]  Alistair John Scarfe.  Development of an autonomous kiwifruit harvester : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Industrial Automation at Massey University, Manawatu, New Zealand. , 2012 .

[55]  Bruce MacDonald,et al.  Robotic kiwifruit harvesting using machine vision, convolutional neural networks, and robotic arms , 2019, Biosystems Engineering.