Evaluating Neural Spatial Interaction Modelling by Bootstrapping

This paper exposes problems of the commonly used technique of splitting the available data in neural spatial interaction modelling into training, validation, and test sets that are held fixed and warns about drawing too strong conclusions from such static splits. Using a bootstrapping procedure, we compare the uncertainty in the solution stemming from the data splitting with model specific uncertainties such as parameter initialization. Utilizing the Austrian interregional telecommunication traffic data and the differential evolution method for solving the parameter estimation task for a fixed topology of the network model [i.e. J = 8] this paper illustrates that the variation due to different resamplings is significantly larger than the variation due to different parameter initializations. This result implies that it is important to not over-interpret a model, estimated on one specific static split of the data.

[1]  Stan Openshaw,et al.  Neural Network, Genetic, and Fuzzy Logic Models of Spatial Interaction , 1998 .

[2]  Blake LeBaron,et al.  A Bootstrap Evaluation of the Effect of Data Splitting on Financial Time Series , 1996, IEEE Trans. Neural Networks.

[3]  Manfred M. Fischer,et al.  ARTIFICIAL NEURAL NETWORKS: A NEW APPROACH TO MODELING INTERREGIONAL TELECOMMUNICATION FLOWS* , 1994 .

[4]  E. Ziegel,et al.  Bootstrapping: A Nonparametric Approach to Statistical Inference , 1993 .

[5]  B Efron,et al.  Statistical Data Analysis in the Computer Age , 1991, Science.

[6]  Heekuck Oh,et al.  Neural Networks for Pattern Recognition , 1993, Adv. Comput..

[7]  B. Efron.  The jackknife, the bootstrap, and other resampling plans , 1987 .

[8]  M M Fischer Tf,et al.  Computational Neural Networks: A New Paradigm for Spatial Analysis , 1998 .

[9]  Robert Tibshirani,et al.  A Comparison of Some Error Estimates for Neural Network Models , 1996, Neural Computation.

[10]  Rainer Storn,et al.  Differential Evolution – A Simple and Efficient Heuristic for global Optimization over Continuous Spaces , 1997, J. Glob. Optim..

[11]  Manfred M. Fischer,et al.  A global search procedure for parameter estimation in neural spatial interaction modelling , 1999 .

[12]  David E. Rumelhart,et al.  Predicting the Future: a Connectionist Approach , 1990, Int. J. Neural Syst..

[13]  Christopher M. Bishop,et al.  Neural networks for pattern recognition , 1995 .

[14]  Stan Openshaw.  Modelling spatial interaction using a neural net , 1993 .

[15]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[16]  Manfred M. Fischer,et al.  A Gobal Search Procedure forParameter Estimation inNeural Spatial Interaction Modelling , 1998 .

[17]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[18]  Hans-Paul Schwefel,et al.  Evolution and optimum seeking , 1995, Sixth-generation computer technology series.