Missing Modalities Imputation via Cascaded Residual Autoencoder

Affordable sensors lead to an increasing interest in acquiring and modeling data with multiple modalities. Learning from multiple modalities has shown to significantly improve performance in object recognition. However, in practice it is common that the sensing equipment experiences unforeseeable malfunction or configuration issues, leading to corrupted data with missing modalities. Most existing multi-modal learning algorithms could not handle missing modalities, and would discard either all modalities with missing values or all corrupted data. To leverage the valuable information in the corrupted data, we propose to impute the missing data by leveraging the relatedness among different modalities. Specifically, we propose a novel Cascaded Residual Autoencoder (CRA) to impute missing modalities. By stacking residual autoencoders, CRA grows iteratively to model the residual between the current prediction and original data. Extensive experiments demonstrate the superior performance of CRA on both the data imputation and the object recognition task on imputed data.

[1]  Jian Sun,et al.  Deep Residual Learning for Image Recognition , 2015, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[2]  Robert Tibshirani,et al.  Spectral Regularization Algorithms for Learning Large Incomplete Matrices , 2010, J. Mach. Learn. Res..

[3]  Jon Atli Benediktsson,et al.  Morphological Attribute Profiles for the Analysis of Very High Resolution Images , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[4]  Gabriele Moser,et al.  Multimodal Classification of Remote Sensing Images: A Review and Future Directions , 2015, Proceedings of the IEEE.

[5]  Elli Angelopoulou,et al.  Supervised multispectral image segmentation with power watersheds , 2012, 2012 19th IEEE International Conference on Image Processing.

[6]  Emmanuel J. Candès,et al.  A Singular Value Thresholding Algorithm for Matrix Completion , 2008, SIAM J. Optim..

[7]  V. Miranda,et al.  Reconstructing Missing Data in State Estimation With Autoencoders , 2012, IEEE Transactions on Power Systems.

[8]  Dieter Fox,et al.  Unsupervised Feature Learning for RGB-D Based Object Recognition , 2012, ISER.

[9]  Antonio J. Plaza,et al.  Fusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature Learning , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[10]  J. R. Jensen,et al.  Hyperspectral Remote Sensing of Vegetation , 2008 .

[11]  William A. Pearlman,et al.  A new, fast, and efficient image codec based on set partitioning in hierarchical trees , 1996, IEEE Trans. Circuits Syst. Video Technol..

[12]  Tshilidzi Marwala,et al.  Missing data: A comparison of neural network and expectation maximization techniques , 2007 .

[13]  Lei Zhang,et al.  Sparse representation or collaborative representation: Which helps face recognition? , 2011, 2011 International Conference on Computer Vision.

[14]  Tshilidzi Marwala,et al.  The use of genetic algorithms and neural networks to approximate missing data in database , 2005, IEEE 3rd International Conference on Computational Cybernetics, 2005. ICCC 2005..

[15]  Yisheng Lv,et al.  A deep learning based approach for traffic data imputation , 2014, 17th International IEEE Conference on Intelligent Transportation Systems (ITSC).

[16]  Juhan Nam,et al.  Multimodal Deep Learning , 2011, ICML.

[17]  Huiling Chen,et al.  Imputing missing values in sensor networks using sparse data representations , 2014, MSWiM '14.

[18]  Yoshua Bengio,et al.  Extracting and composing robust features with denoising autoencoders , 2008, ICML '08.

[19]  Jin Chen,et al.  Multi-modality imagery database for plant phenotyping , 2016, Machine Vision and Applications.

[20]  Emmanuel J. Candès,et al.  Exact Matrix Completion via Convex Optimization , 2009, Found. Comput. Math..

[21]  David Suter,et al.  3D terrestrial LIDAR classifications with super-voxels and multi-scale Conditional Random Fields , 2009, Comput. Aided Des..

[22]  Dieter Fox,et al.  A large-scale hierarchical multi-view RGB-D object dataset , 2011, 2011 IEEE International Conference on Robotics and Automation.

[23]  Geoffrey E. Hinton,et al.  Reducing the Dimensionality of Data with Neural Networks , 2006, Science.

[24]  Quan Pan,et al.  Studies on Hyperspectral Face Recognition in Visible Spectrum With Feature Band Selection , 2010, IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans.

[25]  Andrea Montanari,et al.  Matrix completion from a few entries , 2009, ISIT.

[26]  Takeo Kanade,et al.  Multi-PIE , 2008, 2008 8th IEEE International Conference on Automatic Face & Gesture Recognition.

[27]  Stan Z. Li Markov Random Field Modeling in Image Analysis , 2009, Advances in Pattern Recognition.

[28]  Antonio J. Plaza,et al.  This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination. IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING 1 Spectral–Spatial Hyperspectral Image Segmentation Using S , 2022 .

[29]  A V Kanaev,et al.  Object level HSI-LIDAR data fusion for automated detection of difficult targets. , 2011, Optics express.

[30]  Emmanuel J. Candès,et al.  Matrix Completion With Noise , 2009, Proceedings of the IEEE.

[31]  Andreas Nüchter,et al.  Full Wave Analysis in 3D laser scans for vegetation detection in urban environments , 2011, 2011 XXIII International Symposium on Information, Communication and Automation Technologies.

[32]  S. M. Dhlamini,et al.  Condition Monitoring of HV Bushings in the Presence of Missing Data Using Evolutionary Computing , 2007, ArXiv.

[33]  Qian Du,et al.  Hyperspectral and LiDAR Data Fusion: Outcome of the 2013 GRSS Data Fusion Contest , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[34]  Arif Mahmood,et al.  Hyperspectral Face Recognition With Spatiospectral Information Fusion and PLS Regression , 2015, IEEE Transactions on Image Processing.

[35]  Jeff A. Bilmes,et al.  On Deep Multi-View Representation Learning , 2015, ICML.