A FORTRAN program to produce minimum relative entropy distributions

Abstract Relative entropy minimization is a general approach of inferring a probability density function (pdf) from constraints which do not uniquely determine that density. In this paper, a general purpose computer program written in FORTRAN is provided that produces a univariate pdf from a series of constraints and a prior probability. Some guidelines for the selection of the prior are presented. The FORTRAN code is based on an algorithm that utilizes a Newton–Raphson approach. In addition, we use Gauss–Legendre quadrature for the determination of the integrals, Gauss elimination for matrix solution and a line search for the most optimal Newton step. We present examples of relative entropy minimization involving functions that are geometric moments of a variable x . With a uniform prior p ( x ), classic solutions of statistics are obtained. We also varied the nature of the prior for illustrative purposes. For the case where the constraints resemble powers of x and logarithmic transformations, minimum relative entropy produces the Gamma distribution.

[1]  T. Ulrych,et al.  Minimum relative entropy: Forward probabilistic modeling , 1993 .

[2]  T. Ulrych,et al.  Minimum Relative Entropy Inversion: Theory and Application to Recovering the Release History of a Groundwater Contaminant , 1996 .

[3]  Allan D. Woodbury,et al.  Three-dimensional plume source reconstruction using minimum relative entropy inversion , 1998 .

[4]  T. Skaggs,et al.  Comment on “Minimum relative entropy inversion: Theory and application to recovering the release history of a groundwater contaminant” by Allan D. Woodbury and Tadeusz J. Ulrych , 1998 .

[5]  Kelin Wang,et al.  Minimum relative entropy: Theory and application to surface temperature reconstruction from borehole temperature measurements , 2000 .

[6]  William H. Press,et al.  Numerical recipes in C. The art of scientific computing , 1987 .

[7]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[8]  S. Provencher.  CONTIN: A general purpose constrained regularization program for inverting noisy linear algebraic and integral equations , 1984 .

[9]  S. Kullback,et al.  Information Theory and Statistics , 1959 .

[10]  J. N. Kapur,et al.  Entropy optimization principles with applications , 1992 .

[11]  W. Press,et al.  Numerical Recipes: The Art of Scientific Computing , 1987 .

[12]  L. M. M.-T..  Theory of Probability , 1929, Nature.

[13]  Rodney W. Johnson,et al.  Comments on and correction to 'Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy' (Jan 80 26-37) , 1983, IEEE Trans. Inf. Theory.

[14]  John Skilling,et al.  Maximum Entropy and Bayesian Methods , 1989 .

[15]  W. T. Grandy.  Maximum entropy in action: Buck, Brian and Macaulay, Vincent A., 1991, 220 pp., Clarendon Press, Oxford, £30 pb, ISBN 0-19-8539630 , 1995 .

[16]  T. Ulrych,et al.  Minimum relative entropy and probabilistic inversion in groundwater hydrology , 1998 .

[17]  J. N. Kapur.  Maximum-entropy models in science and engineering , 1992 .

[18]  Brian Borchers,et al.  Comparison of inverse methods for reconstructing the release history of a groundwater contamination source , 2000 .

[19]  S. Provencher.  A constrained regularization method for inverting data represented by linear algebraic or integral equations , 1982 .

[20]  William H. Press,et al.  The Art of Scientific Computing Second Edition , 1998 .

[21]  T. Ulrych,et al.  Reply [to “Comment on ‘Minimum relative entropy inversion: Theory and application to recovering the release history of a groundwater contaminant’ by Allan D. Woodbury and Tadeusz J. Ulrych”] , 1998 .

[22]  A. Mohammad-Djafari.  A Matlab Program to Calculate the Maximum Entropy Distributions , 2001, physics/0111126.

[23]  Rodney W. Johnson,et al.  Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy , 1980, IEEE Trans. Inf. Theory.

[24]  Marvin H. J. Guber.  Bayesian Spectrum Analysis and Parameter Estimation , 1988 .

[25]  Tadeusz J. Ulrych,et al.  A Bayes tour of inversion: A tutorial , 2001 .

[26]  B. Buck,et al.  Linear inversion by the method of maximum entropy , 1989 .