Shelf Auditing Based on Image Classification Using Semi-Supervised Deep Learning to Increase On-Shelf Availability in Grocery Stores

Providing high on-shelf availability (OSA) is a key factor to increase profits in grocery stores. Recently, there has been growing interest in computer vision approaches to monitor OSA. However, the largest and well-known computer vision datasets do not provide annotation for store products, and therefore, a huge effort is needed to manually label products on images. To tackle the annotation problem, this paper proposes a new method that combines two concepts “semi-supervised learning” and “on-shelf availability” (SOSA) for the first time. Moreover, it is the first time that “You Only Look Once” (YOLOv4) deep learning architecture is used to monitor OSA. Furthermore, this paper provides the first demonstration of explainable artificial intelligence (XAI) on OSA. It presents a new software application, called SOSA XAI, with its capabilities and advantages. In the experimental studies, the effectiveness of the proposed SOSA method was verified on image datasets, with different ratios of labeled samples varying from 20% to 80%. The experimental results show that the proposed approach outperforms the existing approaches (RetinaNet and YOLOv3) in terms of accuracy.

[1]  Kaiming He,et al.  Feature Pyramid Networks for Object Detection , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Lei Wang,et al.  Where's the Weet-Bix? , 2007, ACCV.

[3]  Ehtesham Hassan,et al.  Robust visual analysis for planogram compliance problem , 2015, 2015 14th IAPR International Conference on Machine Vision Applications (MVA).

[4]  Eric T. Bradlow,et al.  Structural Estimation of the Effect of Out-of-Stocks , 2010, Manag. Sci..

[5]  Wanqing Li,et al.  Planogram Compliance Checking Based on Detection of Recurring Patterns , 2016, IEEE MultiMedia.

[6]  Rahul Moorthy,et al.  Applying Image Processing for Detecting On-Shelf Availability and Product Positioning in Retail Stores , 2015, WCI '15.

[7]  Francisco Herrera,et al.  Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI , 2020, Inf. Fusion.

[8]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[9]  Katina Michael,et al.  The pros and cons of RFID in supply chain management , 2005, International Conference on Mobile Business (ICMB'05).

[10]  Kaiming He,et al.  Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[11]  Eleonora Bottani,et al.  Monitoring on-shelf availability, out-of-stock and product freshness through RFID in the fresh food supply chain , 2017, International Journal of RF Technologies.

[12]  Kota Iwamoto,et al.  Robust Estimation of Product Amount on Store Shelves from a Surveillance Camera for Improving On-Shelf Availability , 2018, 2018 IEEE International Conference on Imaging Systems and Techniques (IST).

[13]  Trevor Darrell,et al.  Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation , 2013, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[14]  Rakesh Satapathy,et al.  Smart Shelfie -- Internet of Shelves: For Higher On-Shelf Availability , 2015, 2015 IEEE Region 10 Symposium.

[15]  Lei Wang,et al.  Handling Significant Scale Difference for Object Retrieval in a Supermarket , 2009, 2009 Digital Image Computing: Techniques and Applications.

[16]  Tarek R. Besold,et al.  A historical perspective of explainable Artificial Intelligence , 2020, WIREs Data Mining Knowl. Discov..

[17]  Song Liu,et al.  Planogram Compliance Checking Using Recurring Patterns , 2015, 2015 IEEE International Symposium on Multimedia (ISM).

[18]  Ali Farhadi,et al.  YOLOv3: An Incremental Improvement , 2018, ArXiv.

[19]  Hong-Yuan Mark Liao,et al.  YOLOv4: Optimal Speed and Accuracy of Object Detection , 2020, ArXiv.

[20]  Daniel Corsten,et al.  Desperately seeking shelf availability: an examination of the extent, the causes, and the efforts to address retail out‐of‐stocks , 2003 .

[21]  Matthias Dehmer,et al.  Explainable artificial intelligence and machine learning: A reality rooted perspective , 2020, WIREs Data Mining Knowl. Discov..

[22]  Saurav Verma,et al.  On-Shelf Availability in Retailing , 2015 .

[23]  Ebrahim Najafi Kajabad,et al.  People Detection and Finding Attractive Areas by the use of Movement Detection Analysis and Deep Learning Approach , 2019, Procedia Computer Science.

[24]  Ross B. Girshick,et al.  Focal Loss for Dense Object Detection , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Filipe Soares,et al.  Supervised learning for Out-of-Stock detection in panoramas of retail shelves , 2016, 2016 IEEE International Conference on Imaging Systems and Techniques (IST).

[26]  Kota Iwamoto,et al.  Robust Shelf Monitoring Using Supervised Learning for Improving On-Shelf Availability in Retail Stores † , 2019, Sensors.

[27]  Dipti Prasad Mukherjee,et al.  A comprehensive survey on computer vision based approaches for automatic identification of products in retail store , 2019, Image Vis. Comput..

[28]  Swagat Kumar,et al.  Product counting using images with application to robot-based retail stock assessment , 2015, 2015 IEEE International Conference on Technologies for Practical Robot Applications (TePRA).