Hyperparameter optimization of neural network-driven spatial models accelerated using cyber-enabled high-performance computing

ABSTRACT Artificial neural networks (ANNs) have been extensively used for the spatially explicit modeling of complex geographic phenomena. However, because of the complexity of the computational process, there has been an inadequate investigation on the parameter configuration of neural networks. Most studies in the literature from GIScience rely on a trial-and-error approach to select the parameter setting for ANN-driven spatial models. Hyperparameter optimization provides support for selecting the optimal architectures of ANNs. Thus, in this study, we develop an automated hyperparameter selection approach to identify optimal neural networks for spatial modeling. Further, the use of hyperparameter optimization is challenging because hyperparameter space is often large and the associated computational demand is heavy. Therefore, we utilize high-performance computing to accelerate the model selection process. Furthermore, we involve spatial statistics approaches to improve the efficiency of hyperparameter optimization. The spatial model used in our case study is a land price evaluation model in Mecklenburg County, North Carolina, USA. Our results demonstrate that the automated selection approach improves the model-level performance compared with linear regression, and the high-performance computing and spatial statistics approaches are of great help for accelerating and enhancing the selection of optimal neural networks for spatial modeling.

[1]  George Grekousis,et al.  Analyzing High-Risk Emergency Areas with GIS and Neural Networks: The Case of Athens, Greece , 2014 .

[2]  Xiao-Hu Yu,et al.  Efficient Backpropagation Learning Using Optimal Learning Rate and Momentum , 1997, Neural Networks.

[3]  Demetris Stathakis,et al.  How many hidden layers and nodes? , 2009 .

[4]  Charles E. Brown Coefficient of Variation , 1998 .

[5]  G. Matheron Principles of geostatistics , 1963 .

[6]  Chuanrong Zhang,et al.  Spatially non-stationary relationships between urban residential land price and impact factors in Wuhan city, China , 2016 .

[7]  François Laviolette,et al.  Domain-Adversarial Training of Neural Networks , 2015, J. Mach. Learn. Res..

[8]  K. Mera,et al.  Asia's Financial Crisis and the Role of Real Estate , 2000 .

[9]  Kurt Hornik,et al.  Multilayer feedforward networks are universal approximators , 1989, Neural Networks.

[10]  John O. Carter,et al.  Using spatial interpolation to construct a comprehensive archive of Australian climate data , 2001, Environ. Model. Softw..

[11]  Pierre Goovaerts,et al.  Second-Phase Sampling Designs for Non-Stationary Spatial Variables. , 2009, Geoderma.

[12]  P. Moran Notes on continuous stochastic phenomena. , 1950, Biometrika.

[13]  Tom Fawcett,et al.  An introduction to ROC analysis , 2006, Pattern Recognit. Lett..

[14]  Bryan C. Pijanowski,et al.  Calibrating a neural network‐based urban change model for two metropolitan areas of the Upper Midwest of the United States , 2005, Int. J. Geogr. Inf. Sci..

[15]  Chris Eliasmith,et al.  Hyperopt: a Python library for model selection and hyperparameter optimization , 2015 .

[16]  Harry W. Richardson,et al.  What Happened to the CBD-Distance Gradient?: Land Values in a Policentric City , 1989 .

[17]  Donald F. Specht,et al.  Probabilistic neural networks , 1990, Neural Networks.

[18]  Eugene F. Brigham,et al.  The Determinants of Residential Land Values , 1965 .

[19]  Lars Kotthoff,et al.  Auto-WEKA 2.0: Automatic model selection and hyperparameter optimization in WEKA , 2017, J. Mach. Learn. Res..

[20]  Richard J. Beckman,et al.  A Comparison of Three Methods for Selecting Values of Input Variables in the Analysis of Output From a Computer Code , 2000, Technometrics.

[21]  R. Lark Optimized spatial sampling of soil for estimation of the variogram by maximum likelihood , 2002 .

[22]  Juan J. Flores,et al.  The application of artificial neural networks to the analysis of remotely sensed data , 2008 .

[23]  Edzer J. Pebesma,et al.  Multivariable geostatistics in S: the gstat package , 2004, Comput. Geosci..

[24]  H. Ahmad.,et al.  Determining Sample Size for Research Activities , 2017 .

[25]  Stan Openshaw,et al.  Artificial intelligence in geography , 1997 .

[26]  Gustavo E. A. P. A. Batista,et al.  An analysis of four missing data treatment methods for supervised learning , 2003, Appl. Artif. Intell..

[27]  Yoshua Bengio,et al.  Algorithms for Hyper-Parameter Optimization , 2011, NIPS.

[28]  H. Wackernagle,et al.  Multivariate geostatistics: an introduction with applications , 1998 .

[29]  D. Griffith Effective Geographic Sample Size in the Presence of Spatial Autocorrelation , 2005 .

[30]  S. Karsoliya,et al.  Approximating Number of Hidden layer neurons in Multiple Hidden Layer BPNN Architecture , 2012 .

[31]  Peter Kareiva,et al.  Spatial ecology : the role of space in population dynamics and interspecific interactions , 1998 .

[32]  P. D. Heermann,et al.  Classification of multispectral remote sensing data using a back-propagation neural network , 1992, IEEE Trans. Geosci. Remote. Sens..

[33]  George Grekousis,et al.  Modeling urban evolution using neural networks, fuzzy logic and GIS: The case of the Athens metropolitan area , 2013 .

[34]  Paul M. Mather,et al.  The use of backpropagating artificial neural networks in land cover classification , 2003 .

[35]  Bart De Moor,et al.  Hyperparameter Search in Machine Learning , 2015, ArXiv.

[36]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[37]  Jeremy Atack,et al.  “Location, Location, Location!” The Price Gradient for Vacant Urban Land: New York, 1835 to 1900 , 1998 .

[38]  Sucharita Gopal,et al.  Artificial Neural Networks in Geospatial Analysis , 2016 .

[39]  P. Lerman Fitting Segmented Regression Models by Grid Search , 1980 .

[40]  Chris Thornton,et al.  Auto-WEKA : combined selection and hyperparameter optimization of supervised machine learning algorithms , 2014 .

[41]  C. Obled,et al.  Objective analyses and mapping techniques for rainfall fields: An objective comparison , 1982 .

[42]  A. Venables,et al.  The Spatial Economy: Cities, Regions, and International Trade , 2000 .

[43]  Stephen A. Billings,et al.  Non-linear system identification using neural networks , 1990 .

[44]  Barry Wilkinson,et al.  Parallel programming , 1998 .

[45]  S. Blöndal,et al.  House Prices and Economic Activity , 2001 .

[46]  G. Flatman,et al.  Geostatistical strategy for soil sampling: the survey and the census , 1984, Environmental monitoring and assessment.

[47]  R. Yamazaki Empirical testing of real option-pricing models using land price index in Japan , 2001 .

[48]  Michael A. Shanblatt,et al.  A two-phase optimization neural network , 1992, IEEE Trans. Neural Networks.

[49]  Anil K. Bera,et al.  Simple diagnostic tests for spatial dependence , 1996 .

[50]  David D. Cox,et al.  Hyperopt: A Python Library for Optimizing the Hyperparameters of Machine Learning Algorithms , 2013, SciPy.

[51]  Kevin Leyton-Brown,et al.  Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms , 2012, KDD.

[52]  M. D. McKay,et al.  A comparison of three methods for selecting values of input variables in the analysis of output from a computer code , 2000 .

[53]  L. Anselin Local Indicators of Spatial Association—LISA , 2010 .

[54]  R. S. Govindaraju,et al.  Artificial Neural Networks in Hydrology , 2010 .

[55]  Nii O. Attoh-Okine,et al.  Analysis of learning rate and momentum term in backpropagation neural network algorithm trained to predict pavement performance , 1999 .

[56]  Lucila Ohno-Machado,et al.  Logistic regression and artificial neural network classification models: a methodology review , 2002, J. Biomed. Informatics.

[57]  Steven M. LaValle,et al.  On the Relationship between Classical Grid Search and Probabilistic Roadmaps , 2004, Int. J. Robotics Res..

[58]  Anthony Gar-On Yeh,et al.  Neural-network-based cellular automata for simulating multiple land use changes using GIS , 2002, Int. J. Geogr. Inf. Sci..

[59]  Philipp Slusallek,et al.  Introduction to real-time ray tracing , 2005, SIGGRAPH Courses.

[60]  J. Salas,et al.  A COMPARATIVE ANALYSIS OF TECHNIQUES FOR SPATIAL INTERPOLATION OF PRECIPITATION , 1985 .

[61]  C. Willmott Some Comments on the Evaluation of Model Performance , 1982 .

[62]  Wenwu Tang,et al.  Global Sensitivity Analysis of a Large Agent-Based Model of Spatial Opinion Exchange: A Heterogeneous Multi-GPU Acceleration Approach , 2014 .

[63]  G. Metternicht,et al.  Testing the performance of spatial interpolation techniques for mapping soil properties , 2006 .

[64]  N. Lam Spatial Interpolation Methods: A Review , 1983 .

[65]  W. Tobler A Computer Movie Simulating Urban Growth in the Detroit Region , 1970 .

[66]  Vit Vozenilek,et al.  Testing Artificial Neural Network (ANN) for Spatial Interpolation , 2014 .

[67]  Chris Cunningham,et al.  House price uncertainty, timing of development, and vacant land prices: Evidence for real options in Seattle , 2006 .

[68]  S. Huffel,et al.  Logistic Regression and Artificial Neural Networks for Classification of Ovarian Tumors , 2004 .

[69]  Richard Hans Robert Hahnloser,et al.  Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit , 2000, Nature.

[70]  Sheridan Titman,et al.  Do Real Estate Prices and Stock Prices Move Together? An International Analysis , 1998 .

[71]  Sovan Lek,et al.  Applications of artificial neural networks predicting macroinvertebrates in freshwaters , 2007, Aquatic Ecology.

[72]  Bryan C. Pijanowski,et al.  A big data urban growth simulation at a national scale: Configuring the GIS and neural network based Land Transformation Model to run in a High Performance Computing (HPC) environment , 2014, Environ. Model. Softw..

[73]  M. Fortin,et al.  Spatial pattern and ecological analysis , 1989, Vegetatio.

[74]  Rima Tamošiūnienė,et al.  Mass appraisal of residential real estate using multilevel modelling , 2016 .

[75]  S K Thompson,et al.  Spatial sampling. , 1997, Ciba Foundation symposium.

[76]  Alex B. McBratney,et al.  HOW MANY OBSERVATIONS ARE NEEDED FOR REGIONAL ESTIMATION OF SOIL PROPERTIES? , 1983 .

[77]  Vahid Nourani,et al.  Using self-organizing maps and wavelet transforms for space–time pre-processing of satellite precipitation and runoff data in neural network based rainfall–runoff modeling , 2013 .

[78]  John M. Quigley,et al.  Real Estate Prices and Economic Cycles , 2002 .

[79]  B. Pijanowski,et al.  Using neural networks and GIS to forecast land use changes: a Land Transformation Model , 2002 .

[80]  D. G. Krige,et al.  Lognormal-de Wijsian geostatistics for ore evaluation , 1978 .

[81]  Amy J. Ruggles,et al.  An Experimental Comparison of Ordinary and Universal Kriging and Inverse Distance Weighting , 1999 .

[82]  Ahmed Shaker,et al.  Modeling Change-Pattern-Value Dynamics on Land Use: An Integrated GIS and Artificial Neural Networks Approach , 2005, Environmental management.

[83]  Radford M. Neal Pattern Recognition and Machine Learning , 2007, Technometrics.

[84]  Sigrún Andradóttir,et al.  Chapter 20 An Overview of Simulation Optimization via Random Search , 2006, Simulation.

[85]  Xiaodong Li,et al.  A spatial–temporal Hopfield neural network approach for super-resolution land cover mapping with multi-temporal different resolution remotely sensed images , 2014 .

[86]  Daniel Atkins,et al.  Revolutionizing Science and Engineering Through Cyberinfrastructure: Report of the National Science Foundation Blue-Ribbon Advisory Panel on Cyberinfrastructure , 2003 .

[87]  Rémi Bardenet,et al.  Monte Carlo Methods , 2013, Encyclopedia of Social Network Analysis and Mining. 2nd Ed..

[88]  Sotiris B. Kotsiantis,et al.  Supervised Machine Learning: A Review of Classification Techniques , 2007, Informatica.

[89]  J. R. Wallis,et al.  An Approach to Statistical Spatial-Temporal Modeling of Meteorological Fields , 1994 .

[90]  David D. Cox,et al.  Making a Science of Model Search: Hyperparameter Optimization in Hundreds of Dimensions for Vision Architectures , 2013, ICML.

[91]  Martin T. Hagan,et al.  Neural network design , 1995 .

[92]  Xiaojun Yang,et al.  Spatial Interpolation , 2017, Encyclopedia of GIS.

[93]  Robert A. Schowengerdt,et al.  A review and analysis of backpropagation neural networks for classification of remotely-sensed multi-spectral imagery , 1995 .