Neural network based modelling of environmental variables: A systematic approach

Feedforward artificial neural networks (ANNs) that are trained with the back-propagation algorithm are a useful tool for modelling environmental systems. They have already been successfully used to model salinity, nutrient concentrations, air pollution, and algal growth. These successes, coupled with their suitability for modelling complex systems, have resulted in an increase in their popularity and their application in an ever increasing number of areas. They are generally treated as black box models that are able to capture underlying relationships when presented with input and output data. In many instances, little consideration is given to potential input data and the internal workings of ANNs. This can result in inferior model performance and an inability to accurately compare the performance of different ANN models. Back-propagation networks employ a modelling philosophy that is similar to that of statistical methods in the sense that unknown model parameters (i.e., connection weights) are adjusted in order to obtain the best match between a historical set of model inputs and corresponding outputs. Consequently, the principles that are considered good practice in the development of statistical models should be considered. In this paper, a systematic approach to the development of ANN based forecasting models is presented, which is intended to act as a guide for potential and current users of feedforward ANNs that are trained with the back-propagation algorithm. Issues that need to be considered in the model development phase are discussed and ways of addressing them presented. The major areas covered include data transformation, the determination of appropriate model inputs, the determination of an appropriate network geometry, the optimisation of connection weights, and validation of model performance.

[1]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[2]  R. Palmer,et al.  Introduction to the theory of neural computation , 1994, The advanced book program.

[3]  Marija Zlata Boznar,et al.  A neural network-based method for short-term predictions of ambient SO2 concentrations in highly polluted industrial areas of complex terrain , 1993 .

[4]  Qiwen Wang,et al.  Predicting salinity in the chesapeake bay using backpropagation , 1992, Comput. Oper. Res..

[5]  Venugopal G. Krishnapura,et al.  Arma neuron networks for modeling nonlinear dynamical systems , 1997 .

[6]  Frank Fallside,et al.  An adaptive training algorithm for back propagation networks , 1987 .

[7]  Holger R. Maier,et al.  The effect of internal parameters and geometry on the performance of back-propagation neural networks: an empirical study , 1998 .

[8]  J. Faraway,et al.  Time series forecasting with neural networks: a comparative study using the air line data , 2008 .

[9]  Raúl Rojas,et al.  Neural Networks - A Systematic Introduction , 1996 .

[10]  David Haussler,et al.  What Size Net Gives Valid Generalization? , 1989, Neural Computation.

[11]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[12]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[13]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[14]  James M. Ehrman,et al.  Variations in discharge and dissolved organic carbon and nitrogen export from terrestrial basins with changes in climate: A neural network approach , 1996 .

[15]  Ian Flood,et al.  Neural Networks in Civil Engineering. I: Principles and Understanding , 1994 .

[16]  Holger R. Maier,et al.  Understanding the behaviour and optimising the performance of back-propagation neural networks: an empirical study , 1998 .

[17]  J. David Fuller,et al.  Backpropagation in Hydrological Time Series Forecasting , 1994 .

[18]  Thomas Kolarik,et al.  Time series forecasting using neural networks , 1994, APL '94.

[19]  Dominic Maratukulam,et al.  ANNSTLF-a neural-network-based electric load forecasting system , 1997, IEEE Trans. Neural Networks.

[20]  Taha B. M. J. Ouarda,et al.  Comment on “The use of artificial neural networks for the prediction of water quality parameters” by H. R. Maier and G. C. Dandy , 1997 .

[21]  Keith W. Hipel.  Stochastic and statistical methods in hydrology and environmental engineering , 1994 .

[22]  Alan S. Lapedes,et al.  A self-optimizing, nonsymmetrical neural net for content addressable memory and pattern recognition , 1986 .

[23]  Timothy Masters,et al.  Practical neural network recipes in C , 1993 .

[24]  Hava T. Siegelmann,et al.  Computational capabilities of recurrent NARX neural networks , 1997, IEEE Trans. Syst. Man Cybern. Part B.

[25]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1992, Math. Control. Signals Syst..

[26]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[27]  Shin'ichi Tamura,et al.  Capabilities of a four-layered feedforward neural network: four layers versus three , 1997, IEEE Trans. Neural Networks.

[28]  Peter Tiño,et al.  Learning long-term dependencies in NARX recurrent neural networks , 1996, IEEE Trans. Neural Networks.

[29]  D.R. Hush,et al.  Progress in supervised neural networks , 1993, IEEE Signal Processing Magazine.

[30]  Richard M. Golden,et al.  Mathematical Methods for Neural Network Analysis and Design , 1996 .

[31]  A. E. Vries,et al.  Separation of 14C16O and 12C18O by thermal diffusion , 1956 .

[32]  Luís B. Almeida,et al.  Acceleration Techniques for the Backpropagation Algorithm , 1990, EURASIP Workshop.

[33]  L. L. Rogers,et al.  Optimization of groundwater remediation using artificial neural networks with parallel solute transport modeling , 1994 .

[34]  Holger R. Maier,et al.  Use of artificial neural networks for modelling cyanobacteria Anabaena spp. in the River Murray, South Australia , 1998 .

[35]  Larry D. Haugh,et al.  Identification of Dynamic Regression (Distributed Lag) Models Connecting Two Time Series , 1977 .

[36]  Ken-ichi Funahashi,et al.  On the approximate realization of continuous mappings by neural networks , 1989, Neural Networks.

[37]  James P. Ignizio,et al.  Neural networks and operations research: An overview , 1992, Comput. Oper. Res..

[38]  Paul A. Fishwick,et al.  Time series forecasting using neural networks vs. Box- Jenkins methodology , 1991, Simul..

[39]  Brian D. Ripley,et al.  Statistical aspects of neural networks , 1993 .

[40]  Roberto Battiti,et al.  Accelerated Backpropagation Learning: Two Optimization Methods , 1989, Complex Syst..

[41]  Kishan G. Mehrotra,et al.  Forecasting the behavior of multivariate time series using neural networks , 1992, Neural Networks.

[42]  Barry L. Kalman,et al.  Choosing a Sigmoidal Function , 1992 .

[43]  Halbert White,et al.  Learning in Artificial Neural Networks: A Statistical Perspective , 1989, Neural Computation.

[44]  Les E. Atlas,et al.  Recurrent neural networks and robust time series prediction , 1994, IEEE Trans. Neural Networks.

[45]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[46]  Jacques de Villiers,et al.  Backpropagation neural nets with one and two hidden layers , 1993, IEEE Trans. Neural Networks.

[47]  Fernando Gustavo Tomasel,et al.  Prediction of functional characteristics of ecosystems: a comparison of artificial neural networks and regression models , 1997 .

[48]  Holger R. Maier,et al.  Determining Inputs for Neural Network Models of Multivariate Time Series , 1997 .

[49]  H. Maier,et al.  The Use of Artificial Neural Networks for the Prediction of Water Quality Parameters , 1996 .

[50]  George E. P. Box,et al.  Time Series Analysis: Forecasting and Control , 1977 .

[51]  A. Comrie.  Comparing Neural Networks and Regression Models for Ozone Forecasting , 1997 .

[52]  M.H. Hassoun,et al.  Fundamentals of Artificial Neural Networks , 1996, Proceedings of the IEEE.

[53]  Chris Chatfield.  The Analysis of Time Series: Theory and Practice , 1975 .

[54]  Roberto Battiti,et al.  First- and Second-Order Methods for Learning: Between Steepest Descent and Newton's Method , 1992, Neural Computation.

[55]  Brad Warner,et al.  Understanding Neural Networks as Statistical Tools , 1996 .

[56]  Ramazan Gençay,et al.  Nonlinear modelling and prediction with feedforward and recurrent networks , 1997 .

[57]  D. M. Titterington,et al.  Neural Networks: A Review from a Statistical Perspective , 1994 .

[58]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[59]  James V. Hansen,et al.  Neural networks and traditional time series methods: a synergistic combination in state economic forecasts , 1997, IEEE Trans. Neural Networks.

[60]  Chng Eng Siong,et al.  Gradient radial basis function networks for nonlinear and nonstationary time series prediction , 1996, IEEE Trans. Neural Networks.

[61]  Wilfrid S. Kendall,et al.  Networks and Chaos - Statistical and Probabilistic Aspects , 1993 .

[62]  Robert M. Pap,et al.  Handbook of neural computing applications , 1990 .

[63]  Brian D. Ripley,et al.  Neural Networks and Related Methods for Classification , 1994 .

[64]  B. L. Kalman,et al.  Why tanh: choosing a sigmoidal function , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[65]  Holger R. Maier,et al.  Neural Network Models for Forecasting Univariate Time Series , 1996 .