Modeling Multivariate Time Series on Manifolds with Skew Radial Basis Functions

We present an approach for constructing nonlinear empirical mappings from high-dimensional domains to multivariate ranges. We employ radial basis functions and skew radial basis functions for constructing a model using data that are potentially scattered or sparse. The algorithm progresses iteratively, adding a new function at each step to refine the model. The placement of the functions is driven by a statistical hypothesis test that accounts for correlation in the multivariate range variables. The test is applied on training and validation data and reveals nonstatistical or geometric structure when it fails. At each step, the added function is fit to data contained in a spatiotemporally defined local region to determine the parametersin particular, the scale of the local model. The scale of the function is determined by the zero crossings of the autocorrelation function of the residuals. The model parameters and the number of basis functions are determined automatically from the given data, and there is no need to initialize any ad hoc parameters save for the selection of the skew radial basis functions. Compactly supported skew radial basis functions are employed to improve model accuracy, order, and convergence properties. The extension of the algorithm to higher-dimensional ranges produces reduced-order models by exploiting the existence of correlation in the range variable data. Structure is tested not just in a single time series but between all pairs of time series. We illustrate the new methodologies using several illustrative problems, including modeling data on manifolds and the prediction of chaotic time series.

[1]  David S. Broomhead,et al.  A New Approach to Dimensionality Reduction: Theory and Algorithms , 2000, SIAM J. Appl. Math..

[2]  David Lowe,et al.  Practical methods of tracking of nonstationary time series applied to real-world data , 1996, Defense + Commercial Sensing.

[3]  D. Broomhead,et al.  Radial Basis Functions, Multi-Variable Functional Interpolation and Adaptive Networks , 1988 .

[4]  Farmer,et al.  Predicting chaotic time series. , 1987, Physical review letters.

[5]  J. Tenenbaum,et al.  A global geometric framework for nonlinear dimensionality reduction. , 2000, Science.

[6]  Vladimir N. Vapnik,et al.  The Nature of Statistical Learning Theory , 2000, Statistics for Engineering and Information Science.

[7]  B. Nadler,et al.  Diffusion maps, spectral clustering and reaction coordinates of dynamical systems , 2005, math/0503445.

[8]  Stéphane Lafon,et al.  Diffusion maps , 2006 .

[9]  Arta A. Jamshidi,et al.  Towards a Black Box Algorithm for Nonlinear Function Approximation over High-Dimensional Domains , 2007, SIAM J. Sci. Comput..

[10]  Douglas R. Hundley,et al.  Empirical dynamical system reduction II: Neural charts , 1999 .

[11]  Richard A. Davis,et al.  Time Series: Theory and Methods (2nd ed.). , 1992 .

[12]  Simon Haykin,et al.  Regularized radial basis functional networks: theory and applications , 2001 .

[13]  R. L. Hardy.  Multiquadric equations of topography and other irregular surfaces , 1971 .

[14]  Michael Kirby,et al.  Correlation feedback resource allocation RBF , 2001, IJCNN'01. International Joint Conference on Neural Networks. Proceedings (Cat. No.01CH37222).

[15]  Andreas Buja,et al.  Grand tour and projection pursuit , 1995 .

[16]  R. Schaback,et al.  Characterization and construction of radial basis functions , 2001 .

[17]  D. Donoho,et al.  Hessian eigenmaps: Locally linear embedding techniques for high-dimensional data , 2003, Proceedings of the National Academy of Sciences of the United States of America.

[18]  Martin D. Buhmann,et al.  Radial Basis Functions , 2021, Encyclopedia of Mathematical Geosciences.

[19]  Bernhard Schölkopf,et al.  Nonlinear Component Analysis as a Kernel Eigenvalue Problem , 1998, Neural Computation.

[20]  Nikos A. Vlassis,et al.  Non-linear CCA and PCA by Alignment of Local Models , 2003, NIPS.

[21]  F. Takens.  Detecting strange attractors in turbulence , 1981 .

[22]  L. Glass,et al.  Oscillation and chaos in physiological control systems. , 1977, Science.

[23]  Clifford Goodman,et al.  American Society of Mechanical Engineers , 1988 .

[24]  R. Miranda,et al.  Circular Nodes in Neural Networks , 1996, Neural Computation.

[25]  Paramasivan Saratchandran,et al.  Performance evaluation of a sequential minimal radial basis function (RBF) neural network learning algorithm , 1998, IEEE Trans. Neural Networks.

[26]  Visakan Kadirkamanathan,et al.  A Function Estimation Approach to Sequential Learning with Neural Networks , 1993, Neural Computation.

[27]  Lianfen Qian,et al.  Regularized Radial Basis Function Networks: Theory and Applications , 2002, Technometrics.

[28]  Holger Wendland,et al.  Scattered Data Approximation: Conditionally positive definite functions , 2004 .

[29]  Nicolaos B. Karayiannis,et al.  Growing radial basis neural networks: merging supervised and unsupervised learning with network growth techniques , 1997, IEEE Trans. Neural Networks.

[30]  David S. Broomhead,et al.  The Whitney Reduction Network: A Method for Computing Autoassociative Graphs , 2001, Neural Computation.

[31]  John W. Tukey,et al.  A Projection Pursuit Algorithm for Exploratory Data Analysis , 1974, IEEE Transactions on Computers.

[32]  Arta A. Jamshidi,et al.  Examples of Compactly Supported Functions for Radial Basis Approximations , 2006, MLMTA.

[33]  C. C. Homes,et al.  Bayesian Radial Basis Functions of Variable Dimension , 1998, Neural Computation.

[34]  Mikhail Belkin,et al.  Laplacian Eigenmaps for Dimensionality Reduction and Data Representation , 2003, Neural Computation.

[35]  Jooyoung Park,et al.  Approximation and Radial-Basis-Function Networks , 1993, Neural Computation.

[36]  Arta A. Jamshidi.  Modeling Spatio-Temporal Systems with Skew Radial Basis Functions : Theory , Algorithms and Applications , 2008 .

[37]  Yee Whye Teh,et al.  Automatic Alignment of Local Representations , 2002, NIPS.

[38]  David S. Broomhead,et al.  Multivariable Functional Interpolation and Adaptive Networks , 1988, Complex Syst..

[39]  Y Lu,et al.  A Sequential Learning Scheme for Function Approximation Using Minimal Radial Basis Function Neural Networks , 1997, Neural Computation.

[40]  D. Broomhead,et al.  Dimensionality Reduction Using Secant-Based Projection Methods: The Induced Dynamics in Projected Systems , 2005 .

[41]  James P. Crutchfield,et al.  Geometry from a Time Series , 1980 .

[42]  S T Roweis,et al.  Nonlinear dimensionality reduction by locally linear embedding. , 2000, Science.

[43]  Jooyoung Park,et al.  Universal Approximation Using Radial-Basis-Function Networks , 1991, Neural Computation.

[44]  Henry D. I. Abarbanel,et al.  Analysis of Observed Chaotic Data , 1995 .

[45]  Richard A. Davis,et al.  Time Series: Theory and Methods , 2013 .

[46]  Arta A. Jamshidi,et al.  Skew-Radial Basis Function Expansions for Empirical Modeling , 2009, SIAM J. Sci. Comput..

[47]  John C. Platt.  A Resource-Allocating Network for Function Interpolation , 1991, Neural Computation.

[48]  K. Pawelzik,et al.  Optimal Embeddings of Chaotic Attractors from Topological Considerations , 1991 .

[49]  R. N. Desmarais,et al.  Interpolation using surface splines. , 1972 .

[50]  Nicolaos B. Karayiannis,et al.  Growing radial basis neural networks , 1997, Proceedings of International Conference on Neural Networks (ICNN'97).

[51]  Matthew Brand,et al.  Charting a Manifold , 2002, NIPS.

[52]  Geoffrey E. Hinton,et al.  Global Coordination of Local Linear Models , 2001, NIPS.

[53]  H. Abarbanel,et al.  Determining embedding dimension for phase-space reconstruction using a geometrical construction. , 1992, Physical review. A, Atomic, molecular, and optical physics.