Domain Adaptation by Stacked Local Constraint Auto-Encoder Learning

Domain adaptation (DA), a particular case of transfer learning, is an effective technology for learning a discriminative model in scenarios where the data from the training (source) and the testing (target) domains share common class labels but follow different distributions. The differences between domains, called domain shifts, are caused by variations in the acquisition devices and environmental conditions, such as changing illuminations, pose, and collecting-device noises, that are related to a specific domain, denoted as domain-specific noises in this paper. The research on stacked denoising autoencoder (SDA) has demonstrated that noise-robust features could be learned through training a model to reduce the man-made (simulated) noises. However, little research has been conducted to learn the domain-invariant features through training SDA to reduce the domain-specific noises from the real word. In this paper, we propose a novel variant of SDA for DA, called the stacked local constraint auto-encoder (SLC–AE), which aims to learn domain-invariant features through iteratively optimizing the SDA and the low-dimensional manifold. The core idea behind the SLC–AE is that both the source and target samples are corrupted due to the domain-specific noises, and each corrupted sample could be de-noised by calculating the weighted sum of its neighbor samples defined on the intrinsic manifold. Because the neighbor samples on the intrinsic manifold are semantically similar, their weighted sum preserves the generic information and reduces the domain-specific noises. To properly evaluate the performance of the SLC–AE, we conducted extensive experiments using seven benchmark data sets, i.e., MNIST, USPS, COIL20, SYN SIGNS, GTSRB, MSRC and VOC 2007. Compared to twelve different state-of-the-art methods, the experimental results demonstrated that the proposed SLC–AE model made significant improvement over the performance of SDA and achieved the best average performance on the seven data sets.

[1]  Tong Zhang,et al.  Improved Local Coordinate Coding using Local Tangents , 2010, ICML.

[2]  Gabriela Csurka,et al.  Unsupervised Domain Adaptation with Regularized Domain Instance Denoising , 2016, ECCV Workshops.

[3]  Jakob Verbeek,et al.  Heterogeneous Face Recognition with CNNs , 2016, ECCV Workshops.

[4]  Hao Shen,et al.  Trace Quotient Meets Sparsity: A Method for Learning Low Dimensional Image Representations , 2016, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[5]  Angshul Majumdar,et al.  Learning autoencoders with low-rank weights , 2017, 2017 IEEE International Conference on Image Processing (ICIP).

[6]  J. Heckman.  Sample selection bias as a specification error , 1979 .

[7]  Michael I. Jordan,et al.  Learning Transferable Features with Deep Adaptation Networks , 2015, ICML.

[8]  Anton Konushin,et al.  Evaluation of Traffic Sign Recognition Methods Trained on Synthetically Generated Data , 2013, ACIVS.

[9]  Subhransu Maji,et al.  Fast and Accurate Digit Classification , 2009 .

[10]  Khaled A. Harras,et al.  Deep learning and low rank dictionary model for mHealth data classification , 2018, 2018 14th International Wireless Communications & Mobile Computing Conference (IWCMC).

[11]  Antonio Criminisi,et al.  Object categorization by learned universal visual dictionary , 2005, Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1.

[12]  Kate Saenko,et al.  Return of Frustratingly Easy Domain Adaptation , 2015, AAAI.

[13]  Trevor Darrell,et al.  Best Practices for Fine-Tuning Visual Classifiers to New Domains , 2016, ECCV Workshops.

[14]  Yoshua Bengio,et al.  Domain Adaptation for Large-Scale Sentiment Classification: A Deep Learning Approach , 2011, ICML.

[15]  Sameer A. Nene,et al.  Columbia Object Image Library (COIL100) , 1996 .

[16]  Ming Shao,et al.  Deep Robust Encoder Through Locality Preserving Low-Rank Dictionary , 2016, ECCV.

[17]  Brian C. Lovell,et al.  Domain Adaptation on the Statistical Manifold , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[18]  Ivan Laptev,et al.  Learning and Transferring Mid-level Image Representations Using Convolutional Neural Networks , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[19]  Shiguang Shan,et al.  Bi-Shifting Auto-Encoder for Unsupervised Domain Adaptation , 2015, 2015 IEEE International Conference on Computer Vision (ICCV).

[20]  Yao Zhao,et al.  Median filtering detection of small-size image based on CNN , 2018, J. Vis. Commun. Image Represent..

[21]  Jianguo Zhang,et al.  The PASCAL Visual Object Classes Challenge , 2006 .

[22]  Angshul Majumdar,et al.  Discriminative Autoencoder for Feature Extraction: Application to Character Recognition , 2018, Neural Processing Letters.

[23]  Yoshua Bengio,et al.  How transferable are features in deep neural networks? , 2014, NIPS.

[24]  H. Shimodaira,et al.  Improving predictive inference under covariate shift by weighting the log-likelihood function , 2000 .

[25]  Michael I. Jordan,et al.  Deep Transfer Learning with Joint Adaptation Networks , 2016, ICML.

[26]  Trevor Darrell,et al.  Deep Domain Confusion: Maximizing for Domain Invariance , 2014, CVPR 2014.

[27]  Pao-Ta Yu,et al.  Weighted fuzzy mean filters for image processing , 1997, Fuzzy Sets Syst..

[28]  Ke Lu,et al.  Joint Feature Selection and Structure Preservation for Domain Adaptation , 2016, IJCAI.

[29]  Yoshua Bengio,et al.  Generative Adversarial Nets , 2014, NIPS.

[30]  Bianca Zadrozny,et al.  Learning and evaluating classifiers under sample selection bias , 2004, ICML.

[31]  Victor S. Lempitsky,et al.  Neural Codes for Image Retrieval , 2014, ECCV.

[32]  Philip S. Yu,et al.  Transfer Feature Learning with Joint Distribution Adaptation , 2013, 2013 IEEE International Conference on Computer Vision.

[33]  Yuan Shi,et al.  Geodesic flow kernel for unsupervised domain adaptation , 2012, 2012 IEEE Conference on Computer Vision and Pattern Recognition.

[34]  Mikhail Belkin,et al.  Laplacian Eigenmaps and Spectral Techniques for Embedding and Clustering , 2001, NIPS.

[35]  Paul Geladi,et al.  Principal Component Analysis , 1987, Comprehensive Chemometrics.

[36]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[37]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[38]  Kilian Q. Weinberger,et al.  Marginalized Denoising Autoencoders for Domain Adaptation , 2012, ICML.

[39]  Nathalie Japkowicz,et al.  The class imbalance problem: A systematic study , 2002, Intell. Data Anal..

[40]  Jim Jing-Yan Wang,et al.  Supervised Transfer Sparse Coding , 2014, AAAI.

[41]  Hao Hu,et al.  Transfer learning for WiFi-based indoor localization , 2008, AAAI 2008.

[42]  Cheng Wu,et al.  Domain Invariant and Class Discriminative Feature Learning for Visual Domain Adaptation , 2018, IEEE Transactions on Image Processing.

[43]  Trevor Darrell,et al.  DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition , 2013, ICML.

[44]  Rob Fergus,et al.  Visualizing and Understanding Convolutional Networks , 2013, ECCV.

[45]  Ming Shao,et al.  Deep Low-Rank Coding for Transfer Learning , 2015, IJCAI.

[46]  Lina Yao,et al.  Adversarial Collaborative Auto-encoder for Top-N Recommendation , 2019, 2019 International Joint Conference on Neural Networks (IJCNN).

[47]  Ivor W. Tsang,et al.  Domain Adaptation via Transfer Component Analysis , 2009, IEEE Transactions on Neural Networks.

[48]  Johannes Stallkamp,et al.  Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition , 2012, Neural Networks.

[49]  Yihong Gong,et al.  Nonlinear Learning using Local Coordinate Coding , 2009, NIPS.

[50]  Qiang Yang,et al.  A Survey on Transfer Learning , 2010, IEEE Transactions on Knowledge and Data Engineering.

[51]  Mengjie Zhang,et al.  Scatter Component Analysis: A Unified Framework for Domain Adaptation and Domain Generalization , 2015, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[52]  Philip S. Yu,et al.  Transfer Joint Matching for Unsupervised Domain Adaptation , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition.

[53]  Naoto Yokoya,et al.  Learning a Robust Local Manifold Representation for Hyperspectral Dimensionality Reduction , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[54]  Ivor W. Tsang,et al.  Visual Event Recognition in Videos by Learning from Web Data , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[55]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[56]  Victor S. Lempitsky,et al.  Unsupervised Domain Adaptation by Backpropagation , 2014, ICML.

[57]  Wei Wang,et al.  Generalized Autoencoder: A Neural Network Framework for Dimensionality Reduction , 2014, 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops.

[58]  Zhenan Sun,et al.  Aggregating Randomized Clustering-Promoting Invariant Projections for Domain Adaptation , 2019, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[59]  Yun Fu,et al.  Robust Transfer Metric Learning for Image Classification , 2017, IEEE Transactions on Image Processing.