An improved algorithm for the multidimensional moment-constrained maximum entropy problem

The maximum entropy principle is a versatile tool for evaluating smooth approximations of probability density functions with the least bias beyond specified constraints. In the recent paper we introduced new computational framework for the moment-constrained maximum entropy problem in a multidimensional domain, and developed a simple numerical algorithm capable of computing maximum entropy problem in a two-dimensional domain with moment constraints of order up to 4. Here we design an improved numerical algorithm for computing the maximum entropy problem in a two- and higher-dimensional domain with higher order moment constraints. The algorithm features multidimensional orthogonal polynomial basis in the dual space of Lagrange multipliers to achieve numerical stability and rapid convergence of Newton iterations. The new algorithm is found to be capable of solving the maximum entropy problem in the two-dimensional domain with moment constraints of order up to 8, in the three-dimensional domain with moment constraints of order up to 6, and in the four-dimensional domain with moment constraints of order up to 4, corresponding to the total number of moment constraints of 44, 83 and 69, respectively. The two- and higher-dimensional maximum entropy test problems in the current work are based upon long-term statistics of numerical simulation of the real-world geophysical model for wind stress driven oceanic currents such as the Gulf Stream and the Kuroshio.

[1]  Rafail V. Abramov,et al.  A practical computational framework for the multidimensional moment-constrained maximum entropy principle , 2006 .

[2]  Achiya Dax,et al.  A modified Gram–Schmidt algorithm with iterative orthogonalization and column pivoting , 2000 .

[3]  Ximing Wu,et al.  Calculation of maximum entropy densities with application to income distribution , 2003 .

[4]  Prakasa Rao.  Nonparametric functional estimation , 1983 .

[5]  Andrew J. Majda,et al.  Quantifying Uncertainty for Non-Gaussian Ensembles in Complex Systems , 2005, SIAM J. Sci. Comput..

[6]  Andrew J. Majda,et al.  Quantifying predictability through information theory: small sample estimation in a non-Gaussian framework , 2005 .

[7]  R. Haydock,et al.  Comparison of quadrature and termination for estimating the density of states within the recursion method , 1984 .

[8]  Leonard A. Smith,et al.  Evaluating Probabilistic Forecasts Using Information Theory , 2002 .

[9]  Andrew J. Majda,et al.  Information theory and stochastics for multiscale nonlinear systems , 2005 .

[10]  L. Giraud,et al.  When modified Gram–Schmidt generates a well‐conditioned set of vectors , 2002 .

[11]  Yin Zhang,et al.  A Fast Newton Algorithm for Entropy Maximization in Phase Determination , 2001, SIAM Rev..

[12]  Aldo Tagliani,et al.  Hausdorff moment problem and maximum entropy: A unified approach , 1999, Appl. Math. Comput..

[13]  R. Kleeman.  Measuring Dynamical Prediction Utility Using Relative Entropy , 2002 .

[14]  John D. McCalpin,et al.  Phenomenology of the low-frequency variability in a reduced-gravity, quasigeostrophic double-gyre model , 1996 .

[15]  L. Mead,et al.  Maximum entropy in the problem of moments , 1984 .

[16]  Andrew J. Majda,et al.  A framework for predictability through relative entropy , 2002 .

[17]  Gene H. Golub,et al.  Matrix computations , 1983 .

[18]  M. Rozložník,et al.  The loss of orthogonality in the Gram-Schmidt orthogonalization process , 2005 .

[19]  R. Fletcher.  Practical Methods of Optimization , 1988 .

[20]  John D. McCalpin.  The Statistics and Sensitivity of a Double-Gyre Model: The Reduced-Gravity, Quasigeostrophic Case , 1995 .

[21]  S. Griffies,et al.  A Conceptual Framework for Predictability Studies , 1999 .

[22]  Andrew J. Majda,et al.  Information theory and predictability for low-frequency variability , 2005 .

[23]  Dirk Ormoneit,et al.  An efficient algorithm to compute maximum entropy densities , 1999 .

[24]  R. Haydock,et al.  A general terminator for the recursion method , 1985 .

[25]  D. W. Scott,et al.  Multivariate Density Estimation, Theory, Practice and Visualization , 1992 .

[26]  A. K. Bhattacharya,et al.  Maximum entropy and the problem of moments: a stable algorithm. , 2004, Physical review. E, Statistical, nonlinear, and soft matter physics.

[27]  Aldo Tagliani,et al.  Maximum entropy in the finite Stieltjes and Hamburger moment problem , 1994 .

[28]  B. Silverman.  Density estimation for statistics and data analysis , 1986 .

[29]  E. Schmidt.  Über die Auflösung linearer Gleichungen mit Unendlich vielen unbekannten , 1908 .

[30]  I. Turek.  A maximum-entropy approach to the density of states within the recursion method , 1988 .

[31]  Jorge Nocedal,et al.  A Limited Memory Algorithm for Bound Constrained Optimization , 1995, SIAM J. Sci. Comput..