Multiresolution Multimodal Sensor Fusion for Remote Sensing Data With Label Uncertainty

In remote sensing, each sensor can provide complementary or reinforcing information. It is valuable to fuse outputs from multiple sensors to boost overall performance. Previous supervised fusion methods often require accurate labels for each pixel in the training data. However, in many remote-sensing applications, pixel-level labels are difficult or infeasible to obtain. In addition, outputs from multiple sensors often have different resolutions or modalities. For example, rasterized hyperspectral imagery (HSI) presents data in a pixel grid while airborne light detection and ranging (LiDAR) generates dense 3-D point clouds. It is often difficult to directly fuse such multimodal, multiresolution data. To address these challenges, we present a novel multiple instance multiresolution fusion (MIMRF) framework that can fuse multiresolution and multimodal sensor outputs while learning from automatically generated, imprecisely labeled data. Experiments were conducted on the MUUFL Gulfport HSI and LiDAR data set and a remotely sensed soybean and weed data set. Results show improved, consistent performance on scene understanding and agricultural applications when compared to traditional fusion methods.

[1]  Pascal Fua,et al.  SLIC Superpixels Compared to State-of-the-Art Superpixel Methods , 2012, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[2]  Oded Maron,et al.  Multiple-Instance Learning for Natural Scene Classification , 1998, ICML.

[3]  Christophe Labreuche,et al.  The Choquet integral for the aggregation of interval scales in multicriteria decision making , 2003, Fuzzy Sets Syst..

[4]  Louis L. Scharf,et al.  The adaptive coherence estimator: a uniformly most-powerful-invariant adaptive detection statistic , 2005, IEEE Transactions on Signal Processing.

[5]  Aurélien Plyer,et al.  Adaptation and Evaluation of an Optical Flow Method Applied to Coregistration of Forest Remote Sensing Images , 2016, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[6]  Edwin Olson,et al.  Extracting general-purpose features from LIDAR data , 2010, 2010 IEEE International Conference on Robotics and Automation.

[7]  Menas Kafatos,et al.  Wavelet-based hyperspectral and multispectral image fusion , 2001, SPIE Defense + Commercial Sensing.

[8]  Pedram Ghamisi,et al.  Fusion of Hyperspectral and LiDAR Data Using Sparse and Low-Rank Component Analysis , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[9]  P. Gader,et al.  Learning fuzzy measure parameters by logistic LASSO , 2008, NAFIPS 2008 - 2008 Annual Meeting of the North American Fuzzy Information Processing Society.

[10]  Nicolas H. Younan,et al.  Binary fuzzy measures and Choquet integration for multi-source fusion , 2017, 2017 International Conference on Military Technologies (ICMT).

[11]  Jie Shan,et al.  Segmentation and Reconstruction of Polyhedral Building Roofs From Aerial Lidar Point Clouds , 2010, IEEE Transactions on Geoscience and Remote Sensing.

[12]  Pengfei Liu,et al.  Spatial-Hessian-Feature-Guided Variational Model for Pan-Sharpening , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[13]  Stefan Hinz,et al.  Automatic extraction of urban road networks from multi-view aerial imagery , 2003 .

[14]  Changzhe Jiao,et al.  Discriminative Multiple Instance Hyperspectral Target Characterization , 2018, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[15]  Ming-Hsuan Yang,et al.  Visual tracking with online Multiple Instance Learning , 2009, 2009 IEEE Conference on Computer Vision and Pattern Recognition.

[16]  Robert J. Schalkoff,et al.  Digital Image Processing and Computer Vision , 1989 .

[17]  Thomas G. Dietterich,et al.  Solving the Multiple Instance Problem with Axis-Parallel Rectangles , 1997, Artif. Intell..

[18]  János C. Fodor,et al.  Characterization of the ordered weighted averaging operators , 1995, IEEE Trans. Fuzzy Syst..

[19]  P. Gader,et al.  Information fusion and sparsity promotion using choquet integrals , 2007 .

[20]  Paul A. Viola,et al.  Multiple Instance Boosting for Object Detection , 2005, NIPS.

[21]  Janusz Kacprzyk,et al.  Beyond two: theory and applications of multiple-valued logic , 2003 .

[22]  Jocelyn Chanussot,et al.  Fusion of hyperspectral and panchromatic images using multiresolution analysis and nonlinear PCA band reduction , 2011, 2011 IEEE International Geoscience and Remote Sensing Symposium.

[23]  Ming-Hsuan Yang,et al.  Robust Object Tracking with Online Multiple Instance Learning , 2011, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[24]  Roger L. King A challenge for high spatial, spectral, and temporal resolution data fusion , 2000, IGARSS 2000. IEEE 2000 International Geoscience and Remote Sensing Symposium. Taking the Pulse of the Planet: The Role of Remote Sensing in Managing the Environment. Proceedings (Cat. No.00CH37120).

[25]  Paul D. Gader,et al.  Multi-sensor and algorithm fusion with the Choquet integral: applications to landmine detection , 2004, IGARSS 2004. 2004 IEEE International Geoscience and Remote Sensing Symposium.

[26]  Paul D. Gader,et al.  Minimum Classification Error Training for Choquet Integrals With Applications to Landmine Detection , 2008, IEEE Transactions on Fuzzy Systems.

[27]  Xiaoxiao Du,et al.  Possibilistic context identification for SAS imagery , 2015, Defense + Security Symposium.

[28]  Ahmet M. Kondoz,et al.  Fusion of LiDAR and Camera Sensor Data for Environment Sensing in Driverless Vehicles , 2017, ArXiv.

[29]  Xiaoxiao Du,et al.  Multiple Instance Choquet Integral Classifier Fusion and Regression for Remote Sensing Applications , 2018, IEEE Transactions on Geoscience and Remote Sensing.

[30]  Xiaoxiao Du Multiple Instance Choquet Integral for multiresolution sensor fusion , 2017 .

[31]  Liangpei Zhang,et al.  Fusion of Hyperspectral and LiDAR Data for Classification of Cloud-Shadow Mixed Remote Sensed Scene , 2017, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[32]  Michel Grabisch,et al.  A new algorithm for identifying fuzzy measures and its application to pattern recognition , 1995, Proceedings of 1995 IEEE International Conference on Fuzzy Systems..

[33]  Paul Newman,et al.  Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo , 2016, 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).

[34]  S. Süsstrunk,et al.  SLIC Superpixels ? , 2010 .

[35]  Sen Cao,et al.  A Stable Land Cover Patches Method for Automatic Registration of Multitemporal Remote Sensing Images , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[36]  Liangpei Zhang,et al.  An Integrated Framework for the Spatio–Temporal–Spectral Fusion of Remote Sensing Images , 2016, IEEE Transactions on Geoscience and Remote Sensing.

[37]  Kostas Stamatiou,et al.  Combining GeoEye-1 Satellite Remote Sensing, UAV Aerial Imaging, and Geophysical Surveys in Anomaly Detection Applied to Archaeology , 2011, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[38]  James M. Keller,et al.  Multiple Instance Choquet integral for classifier fusion , 2016, 2016 IEEE Congress on Evolutionary Computation (CEC).

[39]  Li Yong,et al.  ADAPTIVE BUILDING EDGE DETECTION BY COMBINING LIDAR DATA AND AERIAL IMAGES , 2008 .

[40]  Lorenzo Bruzzone,et al.  Fusion of Hyperspectral and LIDAR Remote Sensing Data for Classification of Complex Forest Areas , 2008, IEEE Transactions on Geoscience and Remote Sensing.

[41]  D K Smith,et al.  Numerical Optimization , 2001, J. Oper. Res. Soc..

[42]  M. Grabisch The application of fuzzy integrals in multicriteria decision making , 1996 .

[43]  James Llinas,et al.  Handbook of Multisensor Data Fusion , 2001 .

[44]  David A. Clausi,et al.  Weakly Supervised Classification of Remotely Sensed Imagery Using Label Constraint and Edge Penalty , 2017, IEEE Transactions on Geoscience and Remote Sensing.

[45]  Antonio J. Plaza,et al.  Fusion of Hyperspectral and LiDAR Remote Sensing Data Using Multiple Feature Learning , 2015, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[46]  Zhi-Hua Zhou,et al.  Multi-Instance Multi-Label Learning with Application to Scene Classification , 2006, NIPS.

[47]  Christian Wagner,et al.  Efficient Binary Fuzzy Measure Representation and Choquet Integral Learning , 2018, IPMU.

[48]  J. Chanussot,et al.  Hyperspectral Remote Sensing Data Analysis and Future Challenges , 2013, IEEE Geoscience and Remote Sensing Magazine.

[49]  Fang Liu,et al.  Learning Interpolation via Regional Map for Pan-Sharpening , 2015, IEEE Transactions on Geoscience and Remote Sensing.

[50]  Hongtao Yu,et al.  Integration of Heterogeneous Classifiers Based on Choquet Fuzzy Integral , 2015, 2015 7th International Conference on Intelligent Human-Machine Systems and Cybernetics.

[51]  Jean-Luc Marichal,et al.  An axiomatic approach of the discrete Choquet integral as a tool to aggregate interacting criteria , 2000, IEEE Trans. Fuzzy Syst..

[52]  James Llinas,et al.  Handbook of Multisensor Data Fusion : Theory and Practice, Second Edition , 2008 .

[53]  Derek T. Anderson,et al.  Multiple Instance Choquet Integral with Binary Fuzzy Measures for Remote Sensing Classifier Fusion with Imprecise Labels , 2019, 2019 IEEE Symposium Series on Computational Intelligence (SSCI).

[54]  N. L. Johnson,et al.  Continuous Univariate Distributions. , 1995 .

[55]  Leslie M. Collins,et al.  Multiple instance and context dependent learning in hyperspectral data , 2009, 2009 First Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing.

[56]  菅野 道夫,et al.  Theory of fuzzy integrals and its applications , 1975 .

[57]  G. Choquet Theory of capacities , 1954 .

[58]  Jixian Zhang Multi-source remote sensing data fusion: status and trends , 2010 .

[59]  Christine Pohl,et al.  Multisensor image fusion in remote sensing: concepts, methods and applications , 1998 .

[60]  Dražen Odobašić Open Street Map , 2009 .

[61]  Qian Du,et al.  Decision Fusion on Supervised and Unsupervised Classifiers for Hyperspectral Imagery , 2010, IEEE Geoscience and Remote Sensing Letters.

[62]  Manfred Ehlers,et al.  Multi-sensor image fusion for pansharpening in remote sensing , 2010 .

[63]  L.L. Scharf,et al.  Adaptive matched subspace detectors and adaptive coherence estimators , 1996, Conference Record of The Thirtieth Asilomar Conference on Signals, Systems and Computers.

[64]  Dong-Chen He,et al.  Detection of Buildings in Multispectral Very High Spatial Resolution Images Using the Percentage Occupancy Hit-or-Miss Transform , 2014, IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing.

[65]  Michael A. Zatman,et al.  A computationally efficient two-step implementation of the GLRT , 2000, IEEE Trans. Signal Process..

[66]  Bin Wang,et al.  Fusion of Hyperspectral and Multispectral Images: A Novel Framework Based on Generalization of Pan-Sharpening Methods , 2014, IEEE Geoscience and Remote Sensing Letters.

[67]  Mubarak Shah,et al.  Human Action Recognition in Videos Using Kinematic Features and Multiple Instance Learning , 2010, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[68]  Paul D. Gader,et al.  Recognition technology for the detection of buried land mines , 2001, IEEE Trans. Fuzzy Syst..

[69]  Liujuan Cao,et al.  Weakly supervised vehicle detection in satellite images via multi-instance discriminative learning , 2017, Pattern Recognit..

[70]  Oded Maron,et al.  Learning from Ambiguity , 1998 .

[71]  J. Chris McGlone,et al.  Fusion of HYDICE hyperspectral data with panchromatic imagery for cartographic feature extraction , 1999, IEEE Trans. Geosci. Remote. Sens..

[72]  Thomas Hofmann,et al.  Support Vector Machines for Multiple-Instance Learning , 2002, NIPS.

[73]  James M. Keller,et al.  Fundamentals of Computational Intelligence: Neural Networks, Fuzzy Systems, and Evolutionary Computation , 2016 .