Cloud detection from IASI hyperspectral data: a statistical approach based on neural networks

In this work, an investigation of the capability of a statistical cloud detection scheme, implemented through the use of a multilayer feed-forward neural network, is assessed. The whole methodology is applied to a set of IASI L1C spectral radiances, covering the period January 2016-November 2016 and related to Eastern Europe and tropical areas. From a subsampled training dataset where the sky conditions are “certainly” known, we have performed the supervised learning of statistical features of the cloudy- and clear- sky conditions, where truth data have been taken from a cloud mask product of the Advanced Very High-Resolution Radiometer (AVHRR). Also, to improve the neural network classification performances: i) Principal Component Analysis (PCA) of IASI spectra and ii) neural network learning regularization techniques, have been used. Finally, the neural network classification analysis, evaluated during the training with a validation dataset and then with a test dataset, shows very good performance in detecting clouds, with an accuracy of about 93%.

[1]  E. Ben-Dor.  A precaution regarding cirrus cloud detection from airborne imaging spectrometer data using the 1.38 μm water vapor band , 1994 .

[2]  Bin He,et al.  Energy-based cloud detection in multispectral images based on the SVM technique , 2019 .

[3]  Jasper Snoek,et al.  Practical Bayesian Optimization of Machine Learning Algorithms , 2012, NIPS.

[4]  Rich Caruana,et al.  Overfitting in Neural Nets: Backpropagation, Conjugate Gradient, and Early Stopping , 2000, NIPS.

[5]  Christopher D. Barnet,et al.  Hyperspectral Earth Observation from IASI: Five Years of Accomplishments , 2012 .

[6]  Guido Masiello,et al.  Cloud mask via cumulative discriminant analysis applied to satellite infrared observations : scientific basis and initial evaluation , 2014 .

[7]  C. Serio,et al.  Cloud Detection Over Sea Surface by use of Autocorrelation Functions of Upwelling Infrared Spectra in the 800-900-cm(-1) Window Region. , 2000, Applied optics.

[8]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[9]  Jorge Cadima,et al.  Principal component analysis: a review and recent developments , 2016, Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences.

[10]  Vincenzo Cuomo,et al.  Fractality in broken clouds and the scan geometry of new satellite-borne infrared sensors , 2001 .

[11]  Zhe Zhu,et al.  Cloud detection algorithm comparison and validation for operational Landsat data products , 2017 .

[12]  J. van Leeuwen,et al.  Neural Networks: Tricks of the Trade , 2002, Lecture Notes in Computer Science.

[13]  Jeff Heaton,et al.  Ian Goodfellow, Yoshua Bengio, and Aaron Courville: Deep learning , 2017, Genetic Programming and Evolvable Machines.

[14]  C. Serio,et al.  Qualifying IMG tropical spectra for clear sky , 2003 .

[15]  Guigang Zhang,et al.  Deep Learning , 2016, Int. J. Semantic Comput..

[16]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[17]  Nitish Srivastava,et al.  Dropout: a simple way to prevent neural networks from overfitting , 2014, J. Mach. Learn. Res..

[18]  Qin,et al.  Study on the Overfitting of the Artificial Neural Network Forecasting Model , 2004 .

[19]  David M. W. Powers,et al.  Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation , 2011, ArXiv.

[20]  C. Serio,et al.  PCA determination of the radiometric noise of high spectral resolution infrared observations from spectral residuals: Application to IASI , 2018 .

[21]  Timothy Dozat,et al.  Incorporating Nesterov Momentum into Adam , 2016 .

[22]  D. Powers.  Evaluation: From Precision, Recall and F-Factor to ROC, Informedness, Markedness & Correlation , 2008 .

[23]  Lihang Zhou,et al.  AIRS near-real-time products and algorithms in support of operational numerical weather prediction , 2003, IEEE Trans. Geosci. Remote. Sens..

[24]  Eva Borbas,et al.  Diurnal variation in Sahara desert sand emissivity during the dry season from IASI observations , 2014 .

[25]  Carmine Serio,et al.  Inversion for atmospheric thermodynamical parameters of IASI data in the principal components space , 2012 .

[26]  Anestis Antoniadis,et al.  Statistical cloud detection from SEVIRI multispectral images , 2008 .

[27]  Guido Masiello,et al.  Dimensionality‐reduction approach to the thermal radiative transfer equation inverse problem , 2004 .

[28]  W. Paul Menzel,et al.  Cloud Detection of MODIS Multispectral Images , 2014 .

[29]  C. Bohren,et al.  An introduction to atmospheric radiation , 1981 .

[30]  Sepp Hochreiter,et al.  Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) , 2015, ICLR.

[31]  Pietro Mastro,et al.  Characterization of the Observational Covariance Matrix of Hyper-Spectral Infrared Satellite Sensors Directly from Measured Earth Views , 2020, Sensors.

[32]  Anestis Antoniadis,et al.  Cloud Detection: An Assessment Study from the ESA Round Robin Exercise for PROBA-V , 2020, Sensors.

[33]  Anestis Antoniadis,et al.  Technical note: Functional sliced inverse regression to infer temperature, water vapour and ozone from IASI data , 2009 .

[34]  Lance Chun Che Fung,et al.  Classification of Imbalanced Data by Combining the Complementary Neural Network and SMOTE Algorithm , 2010, ICONIP.

[35]  Adam P. Piotrowski,et al.  A comparison of methods to avoid overfitting in neural networks training in the case of catchment runoff modelling , 2013 .

[36]  Tim Hultberg,et al.  Potential for the use of reconstructed IASI radiances in the detection of atmospheric trace gases , 2010 .

[37]  Lutz Prechelt,et al.  Early Stopping - But When? , 2012, Neural Networks: Tricks of the Trade.

[38]  Jihao Yin,et al.  Cloud detection of remote sensing images by deep learning , 2016, 2016 IEEE International Geoscience and Remote Sensing Symposium (IGARSS).

[39]  Ameet Talwalkar,et al.  Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization , 2016, J. Mach. Learn. Res..