Deep-learning-based defective bean inspection with GAN-structured automated labeled data augmentation in coffee industry

In the production process from green beans to coffee bean packages, the defective bean removal (or in short, defect removal) is one of most labor-consuming stages, and many companies investigate the automation of this stage for minimizing human efforts. In this paper, we propose a deep-learning-based defective bean inspection scheme (DL-DBIS), together with a GAN (generative-adversarial network)-structured automated labeled data augmentation method (GALDAM) for enhancing the proposed scheme, so that the automation degree of bean removal with robotic arms can be further improved for coffee industries. The proposed scheme is aimed at providing an effective model to a deep-learning-based object detection module for accurately identifying defects among dense beans. The proposed GALDAM can be used to greatly reduce labor costs, since the data labeling is the most labor-intensive work in this sort of solutions. Our proposed scheme brings two main impacts to intelligent agriculture. First, our proposed scheme is can be easily adopted by industries as human effort in labeling coffee beans are minimized. The users can easily customize their own defective bean model without spending a great amount of time on labeling small and dense objects. Second, our scheme can inspect all classes of defective beans categorized by the SCAA (Specialty Coffee Association of America) at the same time and can be easily extended if more classes of defective beans are added. These two advantages increase the degree of automation in the coffee industry. The prototype of the proposed scheme was developed for studying integrated tests. Testing results of a case study reveal that the proposed scheme can efficiently and effectively generate models for identifying defective beans with accuracy and precision values up to 80%. Appl. Sci. 2019, 9, 4166; doi:10.3390/app9194166 www.mdpi.com/journal/applsci Appl. Sci. 2019, 9, 4166 2 of 26

[1]  Yann LeCun,et al.  Energy-based Generative Adversarial Network , 2016, ICLR.

[2]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[3]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[4]  Yoshinori Fujimura,et al.  High-Throughput Metabolic Profiling of Diverse Green Coffea arabica Beans Identified Tryptophan as a Universal Discrimination Factor for Immature Beans , 2013, PloS one.

[5]  Ali Farhadi,et al.  You Only Look Once: Unified, Real-Time Object Detection , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[6]  Christian E. Portugal-Zambrano,et al.  An approach for improve the recognition of defects in coffee beans using retinex algorithms , 2014, 2014 XL Latin American Computing Conference (CLEI).

[7]  Bernard Widrow,et al.  30 years of adaptive neural networks: perceptron, Madaline, and backpropagation , 1990, Proc. IEEE.

[8]  Rania Hodhod,et al.  AI Cupper: A Fuzzy Expert System for Sensorial Evaluation of Coffee Bean Attributes to Derive Quality Scoring , 2018, IEEE Transactions on Fuzzy Systems.

[9]  Christian E. Portugal-Zambrano,et al.  Computer vision grading system for physical quality evaluation of green coffee beans , 2016, 2016 XLII Latin American Computing Conference (CLEI).

[10]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[11]  Edwin R. Arboleda,et al.  An image processing technique for coffee black beans identification , 2018, 2018 IEEE International Conference on Innovative Research and Development (ICIRD).

[12]  Kari Pulli,et al.  Real-time computer vision with OpenCV , 2012, Commun. ACM.

[13]  Abebe Belay,et al.  Discrimination of Defective (Full Black, Full Sour and Immature) and Nondefective Coffee Beans by Their Physical Properties , 2014 .

[14]  Alexei A. Efros,et al.  Generative Visual Manipulation on the Natural Image Manifold , 2016, ECCV.

[15]  M. Whitworth,et al.  Variability of single bean coffee volatile compounds of Arabica and robusta roasted coffees analysed by SPME-GC-MS , 2018, Food research international.

[16]  Ruifang Ye,et al.  Intelligent defect classification system based on deep learning , 2018 .

[17]  Yuxing Tang,et al.  TUNA-Net: Task-oriented UNsupervised Adversarial Network for Disease Recognition in Cross-Domain Chest X-rays , 2019, MICCAI.

[18]  Luca Mesin,et al.  Control of coffee grinding with Artificial Neural Networks , 2012, The 2012 International Joint Conference on Neural Networks (IJCNN).

[19]  Joachim Rutayisire,et al.  IoT based Coffee quality monitoring and processing system in Rwanda , 2017, 2017 International Conference on Applied System Innovation (ICASI).

[20]  G. Abebe,et al.  Classification of Ethiopian Coffee Beans Using Imaging Techniques , 2013 .

[21]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[22]  Rob Fergus,et al.  Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks , 2015, NIPS.

[23]  Satoshi Tamura,et al.  Classification of Green coffee bean images basec on defect types using convolutional neural network (CNN) , 2017, 2017 International Conference on Advanced Informatics, Concepts, Theory, and Applications (ICAICTA).

[24]  Yuxing Tang,et al.  XLSor: A Robust and Accurate Lung Segmentor on Chest X-Rays Using Criss-Cross Attention and Customized Radiorealistic Abnormalities Generation , 2018, MIDL.

[25]  Yann LeCun,et al.  Disentangling factors of variation in deep representation using adversarial training , 2016, NIPS.

[26]  Wojciech Zaremba,et al.  Improved Techniques for Training GANs , 2016, NIPS.

[27]  Philip S. Yu,et al.  HUOPM: High-Utility Occupancy Pattern Mining , 2018, IEEE Transactions on Cybernetics.

[28]  Philip S. Yu,et al.  A Survey of Utility-Oriented Pattern Mining , 2018, IEEE Transactions on Knowledge and Data Engineering.

[29]  Young-Jin Cha,et al.  Vision-based detection of loosened bolts using the Hough transform and support vector machines , 2016 .

[30]  Hae Yong Kim,et al.  Beans quality inspection using correlation-based granulometry , 2015, Eng. Appl. Artif. Intell..

[31]  Soumith Chintala,et al.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks , 2015, ICLR.

[32]  Philip S. Yu,et al.  A Survey of Parallel Sequential Pattern Mining , 2018, ACM Trans. Knowl. Discov. Data.

[33]  Su Ruan,et al.  Medical Image Synthesis with Context-Aware Generative Adversarial Networks , 2016, MICCAI.

[34]  Jiun-Jian Liaw,et al.  A Fast Randomized Hough Transform for Circle/Circular Arc Recognition , 2010, Int. J. Pattern Recognit. Artif. Intell..

[35]  David Menotti,et al.  Learning Deep Features on Multiple Scales for Coffee Crop Recognition , 2017, 2017 30th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI).

[36]  Gen-Ming Guo,et al.  Quad-Partitioning-Based Robotic Arm Guidance Based on Image Data Processing with Single Inexpensive Camera For Precisely Picking Bean Defects in Coffee Industry , 2019, ACIIDS.

[37]  Tz-Heng Hsu,et al.  Improving Defect Inspection Quality of Deep-Learning Network in Dense Beans by Using Hough Circle Transform for Coffee Industry , 2019, 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC).

[38]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[39]  Teerakiat Kerdcharoen,et al.  Prediction of Acidity Levels of Fresh Roasted Coffees Using E-nose and Artificial Neural Network , 2018, 2018 10th International Conference on Knowledge and Smart Technology (KST).

[40]  Fathurrozi Winjaya,et al.  Identification of cracking sound during coffee roasting using neural network , 2017, 2017 International Seminar on Intelligent Technology and Its Applications (ISITIA).