Neural network uncertainty assessment using Bayesian statistics with application to remote sensing: 3. Network Jacobians

[1] Used for regression fitting, neural network (NN) models can be used effectively to represent highly nonlinear, multivariate functions. In this situation, most emphasis has been on estimating the output errors, but almost no attention has been given to errors associated with the internal structure of the NN model. The complex relationships linking the inputs to the outputs inside the network are the essence of the model and assessing their physical meaning makes all the difference between a “black box” model with small output errors and a physically meaningful model that will provide insight on the problem and will have better generalization properties. Such dependency structures can, for example, be described by the NN Jacobians: they indicate the sensitivity of one output with respect to the inputs of the model. Estimating these Jacobians is essential for many other applications as well. We use a new method of uncertainty estimate developed in the work of Aires [2004] to investigate the robustness of the quantities that characterize the NN structure. A regularization strategy based on principal component analysis is proposed to suppress the multicolinearities that are a major concern when analyzing the internal structure of such a model. The theory is applied to the remote sensing application already presented in the work of Aires [2004] and Aires et al. [2004].

[1]  J. Key,et al.  Classification of merged AVHRR and SMMR Arctic data with neural networks , 1989 .

[2]  Elie Bienenstock,et al.  Neural Networks and the Bias/Variance Dilemma , 1992, Neural Computation.

[3]  C. Rodgers.  Characterization and Error Analysis of Profiles Retrieved From Remote Sounding Measurements , 1990 .

[4]  Ian T. Nabney,et al.  Netlab: Algorithms for Pattern Recognition , 2002 .

[5]  Filipe Aires,et al.  Independent component analysis of multivariate time series: Application to the tropical SST variability , 2000 .

[6]  Filipe Aires.  Problèmes inverses et réseaux de neurones : application à l'interféromètre haute résolution IASI et à l'analyse de séries temporelles , 1999 .

[7]  V. Krasnopolsky,et al.  A neural network as a nonlinear transfer function model for retrieving surface wind speeds from the special sensor microwave imager , 1995 .

[8]  Filipe Aires,et al.  Land surface skin temperatures from a combined analysis of microwave and infrared satellite observations for an all-weather evaluation of the differences between air and skin temperatures , 2003 .

[9]  W. Rossow,et al.  ISCCP Cloud Data Products , 1991 .

[10]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[11]  Filipe Aires,et al.  Joint characterization of vegetation by satellite observations from visible to microwave wavelengths: A sensitivity analysis , 2001 .

[12]  Alain Chedin,et al.  Réseaux de neurones multi-couches pour la restitution de variables thermodynamiques atmosphériques à l'aide de sondeurs verticaux satellitaires , 1993 .

[13]  Geoffrey E. Hinton,et al.  Bayesian Learning for Neural Networks , 1995 .

[14]  Léon Personnaz,et al.  MLPs (Mono-Layer Polynomials and Multi-Layer Perceptrons) for Nonlinear Modeling , 2003, J. Mach. Learn. Res..

[15]  Filipe Aires,et al.  Neural network uncertainty assessment using Bayesian statistics with application to remote sensing: 2. Output errors , 2004 .

[16]  F. Aires,et al.  Neural network uncertainty assessment using Bayesian statistics with application to remote sensing : 1 . Network weights , 2004 .

[17]  Filipe Aires,et al.  Retrieval of Surface and Atmospheric Geophysical Variables over Snow-Covered Land from Combined Microwave and Infrared Satellite Observations , 2003 .

[18]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[19]  F. Aires,et al.  A new neural network approach including first guess for retrieval of atmospheric water vapor, cloud liquid water path, surface temperature, and emissivities over land from satellite microwave observations , 2001 .

[20]  C. Rodgers,et al.  Retrieval of atmospheric temperature and composition from remote measurements of thermal radiation , 1976 .

[21]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[22]  David J. C. MacKay,et al.  A Practical Bayesian Framework for Backpropagation Networks , 1992, Neural Computation.

[23]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[24]  V. Morozov,et al.  Problèmes incorrectement posés: Théorie et applications , 1991 .

[25]  L. Crone,et al.  Statistical applications of a metric on subspaces to satellite meteorology , 1995 .

[26]  I. Jolliffe.  Principal Component Analysis , 2002 .

[27]  Filipe Aires,et al.  Remote sensing from the infrared atmospheric sounding interferometer instrument 1. Compression, denoising, and first-guess retrieval algorithms , 2002 .

[28]  Noëlle A. Scott,et al.  The "weight smoothing" regularization of MLP for Jacobian stabilization , 1999, IEEE Trans. Neural Networks.

[29]  A. Tarantola.  Inverse problem theory : methods for data fitting and model parameter estimation , 1987 .

[30]  G. G. Wilkinson,et al.  Recognition and Inventory of Oceanic Clouds from Satellite Data Using an Artificial Neural Network Technique , 1993 .

[31]  William H. Press,et al.  Numerical Recipes in Fortran 77 , 1992 .

[32]  Richard Lippmann,et al.  Review of Neural Networks for Speech Recognition , 1989, Neural Computation.

[33]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[34]  J. Anthony Gualtieri,et al.  Solving inversion problems with neural networks , 1990, 1990 IJCNN International Joint Conference on Neural Networks.

[35]  Filipe Aires,et al.  Remote sensing from the infrared atmospheric sounding interferometer instrument 2. Simultaneous retrieval of temperature, water vapor, and ozone atmospheric profiles , 2002 .

[36]  Hans Henrik Thodberg,et al.  A review of Bayesian neural networks with an application to near infrared spectroscopy , 1996, IEEE Trans. Neural Networks.

[37]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[38]  Jing M. Chen,et al.  A Vertical Diffusion Scheme to estimate the atmospheric rectifier effect , 2004 .

[39]  Vladimir M. Krasnopolsky,et al.  A Neural Network Multiparameter Algorithm for SSM/I Ocean Retrievals , 2000 .

[40]  Filipe Aires,et al.  Inferring instantaneous, multivariate and nonlinear sensitivities for the analysis of feedback processes in a dynamical system: Lorenz model case‐study , 2003 .

[41]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[42]  C. T. Butler,et al.  Ocean surface wind retrievals from special sensor microwave imager data with neural networks , 1994 .

[43]  Robert A. Schowengerdt,et al.  A review and analysis of backpropagation neural networks for classification of remotely-sensed multi-spectral imagery , 1995 .

[44]  A. N. Tikhonov,et al.  Solutions of ill-posed problems , 1977 .