A Global-Local Artificial Neural Network with Application to Wave Overtopping Prediction

We present a hybrid Radial Basis Function (RBF) - sigmoid neural network with a three-step training algorithm that utilises both global search and gradient descent training. We test the effectiveness of our method using four synthetic datasets and demonstrate its use in wave overtopping prediction. It is shown that the hybrid architecture is often superior to architectures containing neurons of a single type in several ways: lower errors are often achievable using fewer hidden neurons and with less need for regularisation. Our Global-Local Artificial Neural Network (GL-ANN) is also seen to compare favourably with both Perceptron Radial Basis Net (PRBFN) and Regression Tree RBFs.

[1]  Jan Pedersen,et al.  Wave Forces on Crown Walls , 1993 .

[2]  Stephen D. Christman,et al.  Individual Differences in Stroop and Local-Global Processing: A Possible Role of Interhemispheric Interaction , 2001, Brain and Cognition.

[3]  Joachim Diederich,et al.  Survey and critique of techniques for extracting rules from trained artificial neural networks , 1995, Knowl. Based Syst..

[4]  Ah Chung Tsoi,et al.  Noisy Time Series Prediction using Recurrent Neural Networks and Grammatical Inference , 2001, Machine Learning.

[5]  Makarand Deo,et al.  Real time wave forecasting using neural networks , 1998 .

[6]  D. Navon.  Forest before trees: The precedence of global features in visual perception , 1977, Cognitive Psychology.

[7]  S. P. Lloyd,et al.  Least squares quantization in PCM , 1982, IEEE Trans. Inf. Theory.

[8]  Jeff Haberl,et al.  The great energy predictor shootout. II : Measuring retrofit savings , 1998 .

[9]  Holger R. Maier,et al.  Neural networks for the prediction and forecasting of water resource variables: a review of modelling issues and applications , 2000, Environ. Model. Softw..

[10]  J. Ross Quinlan,et al.  Combining Instance-Based and Model-Based Learning , 1993, ICML.

[11]  Thomas M. Cover,et al.  Geometrical and Statistical Properties of Systems of Linear Inequalities with Applications in Pattern Recognition , 1965, IEEE Trans. Electron. Comput..

[12]  David B. Dunson,et al.  Bayesian Data Analysis , 2010 .

[13]  David S. Broomhead,et al.  Multivariable Functional Interpolation and Adaptive Networks , 1988, Complex Syst..

[14]  Rama Chellappa,et al.  Evaluation of pattern classifiers for fingerprint and OCR applications , 1994, Pattern Recognit..

[15]  Babak Hassibi,et al.  Second Order Derivatives for Network Pruning: Optimal Brain Surgeon , 1992, NIPS.

[16]  Rüdiger W. Brause.  edical Analysis and Diagnosis by Neural Networ ks , 2001 .

[17]  P. Werbos,et al.  Beyond Regression : "New Tools for Prediction and Analysis in the Behavioral Sciences , 1974 .

[18]  Various.  Design of seawalls allowing for wave overtopping , 1980 .

[19]  J. Shewchuk.  An Introduction to the Conjugate Gradient Method Without the Agonizing Pain , 1994 .

[20]  William W. Hsieh,et al.  Applying Neural Network Models to Prediction and Data Analysis in Meteorology and Oceanography. , 1998 .

[21]  Zhiye Zhao,et al.  Design of structural modular neural networks with genetic algorithm , 2003 .

[22]  Julia S. J. Yeo,et al.  How Neural Networks Can Help Loan Officers to Make Better Informed Application Decisions , 2003 .

[23]  David J. C. MacKay,et al.  Bayesian Interpolation , 1992, Neural Computation.

[24]  Ananth Ranganathan,et al.  The Levenberg-Marquardt Algorithm , 2004 .

[25]  Simon Haykin,et al.  Neural Networks: A Comprehensive Foundation , 1998 .

[26]  I C G Campbell,et al.  Constructive learning techniques for designing neural network systems , 1998 .

[27]  Xin Yao,et al.  Evolutionary design of artificial neural networks with different nodes , 1996, Proceedings of IEEE International Conference on Evolutionary Computation.

[28]  Geoffrey E. Hinton,et al.  Learning internal representations by error propagation , 1986 .

[29]  J. J. Stoker.  Water Waves: The Mathematical Theory with Applications , 1957 .

[30]  A. W. Jayawardena,et al.  Use of Radial Basis Function Type Artificial Neural Networks for Runoff Simulation , 1998 .

[31]  Donald F. Specht,et al.  A general regression neural network , 1991, IEEE Trans. Neural Networks.

[32]  C. Micchelli.  Interpolation of scattered data: Distance matrices and conditionally positive definite functions , 1986 .

[33]  M. J. D. Powell,et al.  Radial basis functions for multivariable interpolation: a review , 1987 .

[34]  Peggy A. Johnson,et al.  Stream hydrological and ecological responses to climate change assessed with an artificial neural network , 1996 .

[35]  R. Calvo,et al.  Neural Network Prediction of Solar Activity , 1995 .

[36]  J. Shao.  Linear Model Selection by Cross-validation , 1993 .

[37]  Michael I. Jordan,et al.  Hierarchical Mixtures of Experts and the EM Algorithm , 1994 .

[38]  J. Davis.  Statistical Pattern Recognition:Statistical Pattern Recognition , 2003 .

[39]  Linda See,et al.  Applying soft computing approaches to river level forecasting , 1999 .

[40]  Robert J. Abrahart,et al.  Single-Model-Bootstrap Applied to Neural Network Rainfall-Runoff Forecasting , 2001 .

[41]  Adel H. El-Shazly,et al.  The Efficiency of Neural Networks to Model and Predict Monthly Mean Sea Level from Short Spans Applied to Alexandria Tide Gauge , 2005 .

[42]  Ling Qian,et al.  Cartesian Cut Cell Two-Fluid Solver for Hydraulic Flow Problems , 2003 .

[43]  E. Rolls.  High-level vision: Object recognition and visual cognition, Shimon Ullman. MIT Press, Bradford (1996), ISBN 0 262 21013 4 , 1997 .

[44]  Geoffrey G. Towell,et al.  Symbolic knowledge and neural networks: insertion, refinement and extraction , 1992 .

[45]  Nathan Intrator,et al.  A Hybrid Projection Based and Radial Basis Function Architecture , 2000, Multiple Classifier Systems.

[46]  A. N. Tikhonov,et al.  REGULARIZATION OF INCORRECTLY POSED PROBLEMS , 1963 .

[47]  Mohamed S. Kamel,et al.  Modular neural networks: a survey. , 1999, International journal of neural systems.

[48]  Giorgio Valentini,et al.  Ensembles of Learning Machines , 2002, WIRN.

[49]  S. Lawrence,et al.  Function Approximation with Neural Networks and Local Methods: Bias, Variance and Smoothness , 1996 .

[50]  A. Shamseldin.  Application of a neural network technique to rainfall-runoff modelling , 1997 .

[51]  Makarand Deo,et al.  Real‐Time Flood Forecasting Using Neural Networks , 1998 .

[52]  Tomas Hrycej,et al.  Modular Learning in Neural Networks: A Modularized Approach to Neural Network Classification , 1992 .

[53]  Nathan Intrator,et al.  A Hybrid Projection Based and Radial Basis Function Architecture , 2000, Multiple Classifier Systems.

[54]  Kuolin Hsu,et al.  Hydrologic Modelling and Analysis Using A Self- Organizing Linear Output Network , 2002 .

[55]  P. Burman.  A comparative study of ordinary cross-validation, v-fold cross-validation and the repeated learning-testing methods , 1989 .

[56]  Derek M. Causon,et al.  Numerical simulation of wave overtopping of coastal structures using the non-linear shallow water equations , 2000 .

[57]  M. Deo,et al.  Interpolation of wave heights , 2000 .

[58]  M. Kubat,et al.  Decision trees can initialize radial-basis function networks , 1998, IEEE Trans. Neural Networks.

[59]  Richard K. Belew,et al.  Evolving networks: using the genetic algorithm with connectionist learning , 1990 .

[60]  Phillip Ein-Dor,et al.  Attributes of the performance of central processing units: a relative performance prediction model , 1987, CACM.

[61]  Lodewyk F. A. Wessels,et al.  Avoiding False Local Initialization of Minima by Proper Connections , 1992 .

[62]  Zeki Demirbilek,et al.  Journal of Waterway, Port, Coastal, and Ocean Engineering , 1983 .

[63]  W. Pitts,et al.  A Logical Calculus of the Ideas Immanent in Nervous Activity (1943) , 2021, Ideas That Created the Future.

[64]  Ronald Hübner,et al.  Hemispheric Differences in Global/Local Processing Revealed by Same-Different Judgements , 1998 .

[65]  Terry Hedges,et al.  RANDOM WAVE OVERTOPPING OF SIMPLE SEA WALLS: A NEW REGRESSION MODEL. , 1998 .

[66]  Clive G. Mingham,et al.  The applicability of the shallow water equations for modelling violent wave overtopping , 2004 .

[67]  Makarand Deo,et al.  Prediction of breaking waves with neural networks , 2003 .

[68]  Christian W. Dawson,et al.  Hydrological modelling using artificial neural networks , 2001 .

[69]  Bernard Widrow,et al.  Adaptive Signal Processing , 1985 .

[70]  K. Maccorquodale.  Organization of Behavior : A Neuropsychological Theory , 1951 .

[71]  F. Girosi,et al.  Networks for approximation and learning , 1990, Proc. IEEE.

[72]  Ayman Ibrahim,et al.  Hysteresis Sensitive Neural Network for Modeling Rating Curves , 1997 .

[73]  J. Kaas,et al.  The reorganization of somatosensory cortex following peripheral nerve damage in adult and developing mammals. , 1983, Annual review of neuroscience.

[74]  Klaus-Robert Müller,et al.  Statistical Theory of Overtraining - Is Cross-Validation Asymptotically Effective? , 1995, NIPS.

[75]  J J Hopfield,et al.  Neural networks and physical systems with emergent collective computational abilities. , 1982, Proceedings of the National Academy of Sciences of the United States of America.

[76]  George Cybenko,et al.  Approximation by superpositions of a sigmoidal function , 1989, Math. Control. Signals Syst..

[77]  Hadewych Verhaeghe,et al.  Applications of a neural network to predict wave overtopping at Coastal Structures , 2006 .

[78]  William W. Hsieh,et al.  Forecasting regional sea surface temperatures in the tropical Pacific by neural network models, with wind stress and sea level pressure as predictors , 1998 .

[79]  John H. Holland,et al.  Tests on a cell assembly theory of the action of the brain, using a large digital computer , 1956, IRE Trans. Inf. Theory.

[80]  Jooyoung Park,et al.  Approximation and Radial-Basis-Function Networks , 1993, Neural Computation.

[81]  Geoffrey E. Hinton,et al.  Distributed Representations , 1986, The Philosophy of Artificial Intelligence.

[82]  Zhi-Quan Luo,et al.  On the Convergence of the LMS Algorithm with Adaptive Learning Rate for Linear Feedforward Networks , 1991, Neural Computation.

[83]  G. G. Stokes.  "J." , 1890, The New Yale Book of Quotations.

[84]  F ROSENBLATT,et al.  The perceptron: a probabilistic model for information storage and organization in the brain. , 1958, Psychological review.

[85]  Anders Krogh,et al.  A Simple Weight Decay Can Improve Generalization , 1991, NIPS.

[86]  M. Stone.  Cross‐Validatory Choice and Assessment of Statistical Predictions , 1976 .

[87]  John Moody,et al.  Fast Learning in Networks of Locally-Tuned Processing Units , 1989, Neural Computation.

[88]  Tsong-Lin Lee.  Back-propagation neural network for long-term tidal predictions , 2004 .

[89]  Patrick van der Smagt,et al.  Introduction to neural networks , 1995, The Lancet.

[90]  J. Friedman.  Multivariate adaptive regression splines , 1990 .

[91]  J. Freidman,et al.  Multivariate adaptive regression splines , 1991 .

[92]  Rao S. Govindaraju,et al.  Modular Neural Networks for Watershed Runoff , 2000 .

[93]  Shun-ichi Amari,et al.  A Theory of Adaptive Pattern Classifiers , 1967, IEEE Trans. Electron. Comput..

[94]  Masafumi Hagiwara,et al.  Theoretical derivation of momentum term in back-propagation , 1992, [Proceedings 1992] IJCNN International Joint Conference on Neural Networks.

[95]  Robert A. Jacobs,et al.  Increased rates of convergence through learning rate adaptation , 1987, Neural Networks.

[96]  Vasant Honavar,et al.  A neural-network architecture for syntax analysis , 1999, IEEE Trans. Neural Networks.

[97]  C. D. Gelatt,et al.  Optimization by Simulated Annealing , 1983, Science.

[98]  Bernard Widrow,et al.  Adaptive switching circuits , 1988 .

[99]  Peter Rossini,et al.  Using Neural Networks to Estimate Constant Quality House Price Indices , 1999 .

[100]  Padhraic Smyth,et al.  Clustering Using Monte Carlo Cross-Validation , 1996, KDD.

[101]  Zuhair Bandar,et al.  Neural network architectures and overtopping predictions , 2005 .

[102]  P. Besley,et al.  Wave overtopping of seawalls, design and assessment manual , 1998 .

[103]  Terrence J. Sejnowski,et al.  Parallel Networks that Learn to Pronounce English Text , 1987, Complex Syst..

[104]  Raghavan Srinivasan,et al.  PREDICTION OF TWO‐YEAR PEAK STREAM‐DISCHARGES USING NEURAL NETWORKS 1 , 1997 .

[105]  T. Kohonen.  Self-organized formation of topology correct feature maps , 1982 .

[106]  Michael B. Abbott,et al.  Applications of artificial neural networks to the generation of wave equations from hydraulic data , 1999 .

[107]  Simon K. Haslett,et al.  Was the AD 1607 coastal flooding event in the Severn Estuary and Bristol Channel (UK) due to a tsunami , 2002 .

[108]  Dong-Sheng Jeng,et al.  Application of artificial neural networks in tide-forecasting , 2002 .

[109]  N. J. De Vos.  Rainfall-Runoff modelling using artificial neural networks , 2003 .

[110]  Hajime Mase,et al.  NEURAL NETWORK FOR STABILITY ANALYSIS OF RUBBLE-MOUND BREAKWATERS , 1995 .

[111]  S. Charoenseang,et al.  Dynamic self-organized learning for optimizing the complexity growth of radial basis function neural networks , 2002, 2002 IEEE International Conference on Industrial Technology, 2002. IEEE ICIT '02..

[112]  Tim Pullen,et al.  Violent Wave Overtopping at the Coast, When are We Safe? , 2003 .

[113]  C. L. Wilson,et al.  Evaluation of character recognition systems , 1993, Neural Networks for Signal Processing III - Proceedings of the 1993 IEEE-SP Workshop.

[114]  J. von Neumann,et al.  Probabilistic Logic and the Synthesis of Reliable Organisms from Unreliable Components , 1956 .

[115]  Vasant Honavar,et al.  Evolutionary Design of Neural Architectures -- A Preliminary Taxonomy and Guide to Literature , 1995 .

[116]  O. Makarynskyy,et al.  Improving wave predictions with artificial neural networks , 2004 .

[117]  Xin Yao,et al.  A review of evolutionary artificial neural networks , 1993, Int. J. Intell. Syst..

[118]  Susan A. Murphy,et al.  Monographs on statistics and applied probability , 1990 .

[119]  Leopoldo Franco,et al.  WAVE OVERTOPPING ON RUBBLE MOUND BREAKWATERS , 1988 .

[120]  Ke Chen,et al.  A self-generating modular neural network architecture for supervised learning , 1997, Neurocomputing.

[121]  Michael Kearns,et al.  A Bound on the Error of Cross Validation Using the Approximation and Estimation Rates, with Consequences for the Training-Test Split , 1995, Neural Computation.

[122]  D. Rubinfeld,et al.  Hedonic housing prices and the demand for clean air , 1978 .

[123]  Sönke Johannes,et al.  Hierarchical visual stimuli: electrophysiological evidence for separate left hemispheric global and local processing mechanisms in humans , 1996, Neuroscience Letters.

[124]  P. A. D. Bird,et al.  The influence of air and scale on wave impact pressures , 2001 .

[125]  I. Johnstone,et al.  Projection-Based Approximation and a Duality with Kernel Methods , 1989 .

[126]  John E. Moody,et al.  Fast Learning in Multi-Resolution Hierarchies , 1988, NIPS.

[127]  Christoph Schommer.  Incremental Discovery of Association Rules with Dynamic Neural Cells , 2004 .

[128]  Derek M. Causon,et al.  On the Validity of the Shallow Water Equations for Violent Wave Over topping , 2002 .

[129]  Dimitri P. Solomatine,et al.  On the encapsulation of numerical-hydraulic models in artificial neural network , 1999 .

[130]  Wenrui Huang,et al.  Application of an artificial neural network to predict tidal currents in an inlet , 2003 .

[131]  Mohammad Bagher Menhaj,et al.  Training feedforward networks with the Marquardt algorithm , 1994, IEEE Trans. Neural Networks.

[132]  Jason Smith,et al.  Neural-Network Models of Rainfall-Runoff Process , 1995 .

[133]  Derek M. Causon,et al.  Numerical solutions of the shallow water equations with discontinuous bed topography , 2002 .

[134]  Sei-Wang Chen,et al.  Automatic license plate recognition , 2004, IEEE Transactions on Intelligent Transportation Systems.

[135]  J. A. Battjes,et al.  Wave height distributions on shallow foreshores , 2000 .

[136]  C. Malsburg,et al.  How patterned neural connections can be set up by self-organization , 1976, Proceedings of the Royal Society of London. Series B. Biological Sciences.

[137]  J. Mark.  Introduction to radial basis function networks , 1996 .