Online Continual Learning For Visual Food Classification

Food image classification is challenging for real-world applications since existing methods require static datasets for training and are not capable of learning from sequentially available new food images. Online continual learning aims to learn new classes from data stream by using each new data only once without forgetting the previously learned knowledge. However, none of the existing works target food image analysis, which is more difficult to learn incrementally due to its high intra-class variation with the unbalanced and unpredictable characteristics of future food-class distribution. In this paper, we address these issues by introducing (1) a novel clustering based exemplar selection algorithm to store the most representative data belonging to each learned food for knowledge replay, and (2) an effective online learning regime using balanced training batch along with the knowledge distillation on augmented exemplars to maintain the model performance on all learned classes. Our method is evaluated on a challenging large scale food image database, Food-1K1, by varying the number of newly added food classes. Our results show significant improvements compared with existing state-of-the-art online continual learning methods, showing great potential to achieve lifelong learning for food image classification in real world.

[1]  Gabriela Csurka,et al.  Distance-Based Image Classification: Generalizing to New Classes at Near-Zero Cost , 2013, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Derek Hoiem,et al.  Learning without Forgetting , 2016, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[3]  Yandong Guo,et al.  Large Scale Incremental Learning , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  Yen-Cheng Liu,et al.  Re-evaluating Continual Learning Scenarios: A Categorization and Case for Strong Baselines , 2018, ArXiv.

[5]  Zhiling Wang,et al.  ISIA Food-500: A Dataset for Large-Scale Food Recognition via Stacked Global-Local Attention Network , 2020, ACM Multimedia.

[6]  Max Welling,et al.  Herding dynamical weights to learn , 2009, ICML '09.

[7]  Keiji Yanai,et al.  Image Recognition of 85 Food Categories by Feature Fusion , 2010, 2010 IEEE International Symposium on Multimedia.

[8]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[9]  Fengqing Zhu,et al.  Towards Learning Food Portion From Monocular Images With Cross-Domain Feature Adaptation , 2021, 2021 IEEE 23rd International Workshop on Multimedia Signal Processing (MMSP).

[10]  Christoph H. Lampert,et al.  iCaRL: Incremental Classifier and Representation Learning , 2016, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[11]  Fengqing Zhu,et al.  An End-to-End Food Image Analysis System , 2021, Electronic Imaging.

[12]  Fei Yin,et al.  Prototype Augmentation and Self-Supervision for Incremental Learning , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[13]  Fengqing Zhu,et al.  Visual Aware Hierarchy Based Food Recognition , 2020, ICPR Workshops.

[14]  C. J. Boushey,et al.  New mobile methods for dietary assessment: review of image-assisted and image-based dietary assessment methods , 2016, Proceedings of the Nutrition Society.

[15]  Jiangpeng He,et al.  Unsupervised Continual Learning Via Pseudo Labels , 2021, CSSL.

[16]  Marc'Aurelio Ranzato,et al.  On Tiny Episodic Memories in Continual Learning , 2019 .

[17]  Davide Maltoni,et al.  Continuous Learning in Single-Incremental-Task Scenarios , 2018, Neural Networks.

[18]  David G. Lowe,et al.  Object recognition from local scale-invariant features , 1999, Proceedings of the Seventh IEEE International Conference on Computer Vision.

[19]  Nathan D. Cahill,et al.  Memory Efficient Experience Replay for Streaming Learning , 2018, 2019 International Conference on Robotics and Automation (ICRA).

[20]  Geoffrey E. Hinton,et al.  Visualizing Data using t-SNE , 2008 .

[21]  Keiji Yanai,et al.  DeepFoodCam: A DCNN-based Real-time Mobile Food Recognition System , 2016, MADiMa @ ACM Multimedia.

[22]  Fengqing Zhu,et al.  Multi-task Image-Based Dietary Assessment for Food Recognition and Portion Size Estimation , 2020, 2020 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR).

[23]  Adrian Popescu,et al.  IL2M: Class Incremental Learning With Dual Memory , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[24]  Geoffrey E. Hinton,et al.  Distilling the Knowledge in a Neural Network , 2015, ArXiv.

[25]  E. Delp,et al.  The Most Frequently Consumed and the Largest Energy Contributing Foods of U.S. Insulin Takers Using NHANES 2009–2016 , 2021 .

[26]  Gian Luca Foresti,et al.  Wide-Slice Residual Networks for Food Recognition , 2016, 2018 IEEE Winter Conference on Applications of Computer Vision (WACV).

[27]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[28]  Jeffrey Scott Vitter,et al.  Random sampling with a reservoir , 1985, TOMS.

[29]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[30]  Shutao Xia,et al.  Maintaining Discrimination and Fairness in Class Incremental Learning , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[31]  Kibok Lee,et al.  Overcoming Catastrophic Forgetting With Unlabeled Data in the Wild , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[32]  Simone Calderara,et al.  Conditional Channel Gated Networks for Task-Aware Continual Learning , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[33]  Michael McCloskey,et al.  Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem , 1989 .

[34]  Philip H. S. Torr,et al.  GDumb: A Simple Approach that Questions Our Progress in Continual Learning , 2020, ECCV.

[35]  Shuqiang Jiang,et al.  Large Scale Visual Food Recognition , 2021, ArXiv.

[36]  Jeff Johnson,et al.  An Evaluation of Large-scale Methods for Image Instance and Class Discovery , 2017, ACM Multimedia.

[37]  Cordelia Schmid,et al.  End-to-End Incremental Learning , 2018, ECCV.

[38]  Luca Antiga,et al.  Automatic differentiation in PyTorch , 2017 .

[39]  Fengqing Zhu,et al.  Incremental Learning in Online Scenario , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[40]  Michele Merler,et al.  Learning to Make Better Mistakes: Semantics-aware Visual Food Recognition , 2016, ACM Multimedia.

[41]  William W. Cohen,et al.  Power Iteration Clustering , 2010, ICML.

[42]  Makoto Ogawa,et al.  Food Detection and Recognition Using Convolutional Neural Network , 2014, ACM Multimedia.

[43]  Dahua Lin,et al.  Learning a Unified Classifier Incrementally via Rebalancing , 2019, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).