NP-Match: When Neural Processes meet Semi-Supervised Learning

Semi-supervised learning (SSL) has been widely explored in recent years, and it is an effective way of leveraging unlabeled data to reduce the reliance on labeled data. In this work, we adjust neural processes (NPs) to the semi-supervised image classification task, resulting in a new method named NP-Match. NP-Match is suited to this task for two reasons. Firstly, NP-Match implicitly compares data points when making predictions, and as a result, the prediction of each unlabeled data point is affected by the labeled data points that are similar to it, which improves the quality of pseudo-labels. Secondly, NP-Match is able to estimate uncertainty that can be used as a tool for selecting unlabeled samples with reliable pseudo-labels. Compared with uncertainty-based SSL methods implemented with Monte Carlo (MC) dropout, NP-Match estimates uncertainty with much less computational overhead, which can save time at both the training and the testing phases. We conducted extensive experiments on four public datasets, and NP-Match outperforms state-of-the-art (SOTA) results or achieves competitive results on them, which shows the effectiveness of NP-Match and its potential for SSL.

[1]  T. Shinozaki,et al.  FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling , 2021, NeurIPS.

[2]  Yinghuan Shi,et al.  Inconsistency-Aware Uncertainty Estimation for Semi-Supervised Medical Image Segmentation , 2021, IEEE Transactions on Medical Imaging.

[3]  Gholamreza Haffari,et al.  All Labels Are Not Created Equal: Enhancing Semi-supervision via Label Grouping and Co-training , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[4]  R. Nevatia,et al.  SimPLE: Similar Pseudo Label Exploitation for Semi-Supervised Classification , 2021, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Mubarak Shah,et al.  In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning , 2021, ICLR.

[6]  Richard E. Turner,et al.  The Gaussian Neural Process , 2021, ArXiv.

[7]  Jiebo Luo,et al.  EnAET: A Self-Trained Framework for Semi-Supervised and Supervised Learning With Ensemble Transformations , 2020, IEEE Transactions on Image Processing.

[8]  Omesh Tickoo,et al.  Improving model calibration with accuracy versus uncertainty optimization , 2020, NeurIPS.

[9]  Junnan Li,et al.  CoMatch: Semi-supervised Learning with Contrastive Graph Regularization , 2020, 2021 IEEE/CVF International Conference on Computer Vision (ICCV).

[10]  Chek Sing Teo,et al.  Grasping Detection Network with Uncertainty Estimation for Confidence-Driven Semi-Supervised Domain Adaptation , 2020, 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[11]  Yee Whye Teh,et al.  Bootstrapping Neural Processes , 2020, NeurIPS.

[12]  Sungjin Ahn,et al.  Robustifying Sequential Neural Processes , 2020, ICML.

[13]  Max-Heinrich Laves,et al.  Calibration of Model Uncertainty for Dropout Variational Inference , 2020, ArXiv.

[14]  Vishal M. Patel,et al.  Syn2Real Transfer Learning for Image Deraining Using Gaussian Processes , 2020, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[15]  Yao Hu,et al.  Uncertainty Aware Graph Gaussian Process for Semi-Supervised Learning , 2020, AAAI.

[16]  Quoc V. Le,et al.  Meta Pseudo Labels , 2020, 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]  David Berthelot,et al.  FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence , 2020, NeurIPS.

[18]  F. Nielsen.  On a Generalization of the Jensen–Shannon Divergence and the Jensen–Shannon Centroid , 2019, Entropy.

[19]  Nicholas Carlini,et al.  ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring , 2019, ArXiv.

[20]  Richard E. Turner,et al.  Convolutional Conditional Neural Processes , 2019, ICLR.

[21]  Wenshuo Wang,et al.  Recurrent Attentive Neural Process for Sequential Data , 2019, ArXiv.

[22]  Gadi Wollstein,et al.  Uncertainty Guided Semi-supervised Segmentation of Retinal Layers in OCT Images , 2019, MICCAI.

[23]  Quoc V. Le,et al.  Randaugment: Practical automated data augmentation with a reduced search space , 2019, 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).

[24]  Chi-Wing Fu,et al.  Uncertainty-aware Self-ensembling Model for Semi-supervised 3D Left Atrium Segmentation , 2019, MICCAI.

[25]  Klamer Schutte,et al.  The Functional Neural Process , 2019, NeurIPS.

[26]  Sebastian Nowozin,et al.  Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes , 2019, NeurIPS.

[27]  Sungjin Ahn,et al.  Sequential Neural Processes , 2019, NeurIPS.

[28]  Ian Walker,et al.  Graph Convolutional Gaussian Processes , 2019, ICML.

[29]  Alexander Kolesnikov,et al.  S4L: Self-Supervised Semi-Supervised Learning , 2019, 2019 IEEE/CVF International Conference on Computer Vision (ICCV).

[30]  David Berthelot,et al.  MixMatch: A Holistic Approach to Semi-Supervised Learning , 2019, NeurIPS.

[31]  Quoc V. Le,et al.  Unsupervised Data Augmentation for Consistency Training , 2019, NeurIPS.

[32]  S. Shreve,et al.  Stochastic differential equations , 1955, Mathematical Proceedings of the Cambridge Philosophical Society.

[33]  Yin Cheng Ng,et al.  Bayesian Semi-supervised Learning with Graph Gaussian Processes , 2018, NeurIPS.

[34]  Yee Whye Teh,et al.  Conditional Neural Processes , 2018, ICML.

[35]  Zhi-Hua Zhou,et al.  Tri-net for Semi-Supervised Deep Learning , 2018, IJCAI.

[36]  Stefano Ermon,et al.  Semi-supervised Deep Kernel Learning: Regression with Unlabeled Data by Minimizing Predictive Variance , 2018, NeurIPS.

[37]  Bo Wang,et al.  Deep Co-Training for Semi-Supervised Image Recognition , 2018, ECCV.

[38]  Kilian Q. Weinberger,et al.  On Calibration of Modern Neural Networks , 2017, ICML.

[39]  Alex Kendall,et al.  What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision? , 2017, NIPS.

[40]  Timo Aila,et al.  Temporal Ensembling for Semi-Supervised Learning , 2016, ICLR.

[41]  Frank Hutter,et al.  SGDR: Stochastic Gradient Descent with Warm Restarts , 2016, ICLR.

[42]  Tolga Tasdizen,et al.  Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning , 2016, NIPS.

[43]  Nikos Komodakis,et al.  Wide Residual Networks , 2016, BMVC.

[44]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[45]  Sergey Ioffe,et al.  Rethinking the Inception Architecture for Computer Vision , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[46]  Andrew Gordon Wilson,et al.  Deep Kernel Learning , 2015, AISTATS.

[47]  Zoubin Ghahramani,et al.  Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning , 2015, ICML.

[48]  Thomas Brox,et al.  U-Net: Convolutional Networks for Biomedical Image Segmentation , 2015, MICCAI.

[49]  Milos Hauskrecht,et al.  Obtaining Well Calibrated Probabilities Using Bayesian Binning , 2015, AAAI.

[50]  Philip Bachman,et al.  Learning with Pseudo-Ensembles , 2014, NIPS.

[51]  Dumitru Erhan,et al.  Going deeper with convolutions , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[52]  Andrew Zisserman,et al.  Very Deep Convolutional Networks for Large-Scale Image Recognition , 2014, ICLR.

[53]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[54]  Honglak Lee,et al.  An Analysis of Single-Layer Networks in Unsupervised Feature Learning , 2011, AISTATS.

[55]  Zhi-Hua Zhou,et al.  Towards Making Unlabeled Data Never Hurt , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[56]  Wei Liu,et al.  Large Graph Construction for Scalable Semi-Supervised Learning , 2010, ICML.

[57]  Frank Nielsen,et al.  Statistical exponential families: A digest with flash cards , 2009, ArXiv.

[58]  Fei-Fei Li,et al.  ImageNet: A large-scale hierarchical image database , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[59]  Wei Chu,et al.  Semi-Supervised Gaussian Process Classifiers , 2007, IJCAI.

[60]  Yan Wang,et al.  Tripled-Uncertainty Guided Mean Teacher Model for Semi-supervised Medical Image Segmentation , 2021, MICCAI.

[61]  Dong-Hyun Lee,et al.  Pseudo-Label : The Simple and Efficient Semi-Supervised Learning Method for Deep Neural Networks , 2013 .

[62]  Alex Krizhevsky,et al.  Learning Multiple Layers of Features from Tiny Images , 2009 .

[63]  P. K. Jain.  CONVEX FUNCTIONS AND THEIR APPLICATIONS , 1968 .

[64]  A. Krizhevsky.  ImageNet Classification with Deep Convolutional Neural Networks , 2022 .