On the recovery of joint distributions from limited information

The paper considers the role of entropy and other information theoretic concepts in the description and formation of joint distributions for m random variables. The discussion includes a review of methods for constructing discrete and continuous joint distributions from the component marginal distributions. We then propose a minimum cross-entropy approach that recovers continuous joint distributions from the joint and marginal moments and the marginal densities. The large-sample properties of the associated estimator are outlined, and a simple demonstration problem that highlights the advantages and drawbacks of the proposed method is presented.

[1]  H. Ohta,et al.  A Test for Normality Based on Kullback—Leibler Information , 1989 .

[2]  S. Kullback,et al.  The Information in Contingency Tables , 1980 .

[3]  E. Ziegel.  COMPSTAT: Proceedings in Computational Statistics , 1988 .

[4]  Allan W. Gray,et al.  An Applied Procedure for Estimating and Simulating Multivariate Empirical (MVE) Probability Distributions In Farm-Level Risk Assessment and Policy Analysis , 2000, Journal of Agricultural and Applied Economics.

[5]  C. R. Taylor.  Two Practical Procedures for Estimating Multivariate Nonnormal Probability Density Functions , 1990 .

[6]  B. R. Crain.  An Information Theoretic Approach to Approximating a Probability Distribution , 1977 .

[7]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[8]  S. Kotz,et al.  Correlation Structure in Iterated Farlie-Gumbel-Morgenstern Distributions. , 1984 .

[9]  L. Devroye.  A Course in Density Estimation , 1987 .

[10]  L. Cobb,et al.  Estimation and Moment Recursion Relations for Multimodal Distributions of the Exponential Family , 1983 .

[11]  Kenneth J. Koehler,et al.  A strategy for constructing multivariate distributions , 1995 .

[12]  M. Jansen,et al.  Maximum entropy distributions with prescribed marginals and normal score correlations , 1997 .

[13]  N. L. Johnson,et al.  On some generalized farlie-gumbel-morgenstern distributions-II regression, correlation and further generalizations , 1977 .

[14]  Dou Long,et al.  A family of bivariate densities constructed from marginals , 1995 .

[15]  I. Csiszár.  $I$-Divergence Geometry of Probability Distributions and Minimization Problems , 1975 .

[16]  R. Mittelhammer.  Mathematical Statistics for Economics and Business , 1996 .

[17]  Stuart A. Klugman,et al.  Fitting bivariate loss distributions with copulas , 1999 .

[18]  M. E. Johnson,et al.  A Bivariate Distribution Family with Specified Marginals , 1981 .

[19]  W. Deming,et al.  On a Least Squares Adjustment of a Sampled Frequency Table When the Expected Marginal Totals are Known , 1940 .

[20]  Norman L. Johnson,et al.  On some generalized farlie-gumbel-morgenstern distributions , 1975 .

[21]  S. Kullback,et al.  Contingency tables with given marginals. , 1968, Biometrika.

[22]  I. Olkin,et al.  Families of Multivariate Distributions , 1988 .

[23]  Douglas J. Miller,et al.  Maximum entropy econometrics: robust estimation with limited data , 1996 .

[24]  S. Kotz,et al.  A New Approach to Dependence in Multivariate Distributions , 1991 .

[25]  Dirk Ormoneit,et al.  An efficient algorithm to compute maximum entropy densities , 1999 .

[26]  Shelby J. Haberman,et al.  Adjustment by Minimum Discriminant Information , 1984 .

[27]  M. Piedmonte,et al.  A Method for Generating High-Dimensional Multivariate Binary Variates , 1991 .

[28]  Jose Luis Beltran Guerrero,et al.  Multivariate mutual information , 1994 .

[29]  K. Marton.  Bounding $\bar{d}$-distance by informational divergence: a method to prove measure concentration , 1996 .

[30]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[31]  C. Klaassen,et al.  Efficient estimation in the bivariate normal copula model: normal margins are least favourable , 1997 .

[32]  I. Good.  Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Tables , 1963 .

[33]  Samuel Kotz,et al.  Advances in Probability Distributions with Given Marginals , 1991 .

[34]  Amiel Feinstein,et al.  Information and information stability of random variables and processes , 1964 .

[35]  G. Dall’aglio,et al.  Frechet Classes: The Beginnings , 1991 .

[36]  C. Genest,et al.  The Joy of Copulas: Bivariate Distributions with Uniform Marginals , 1986 .

[37]  Richard A. Highfield,et al.  Calculation of maximum entropy distributions and approximation of marginalposterior distributions , 1988 .

[38]  J. Berger.  Statistical Decision Theory and Bayesian Analysis , 1988 .

[39]  W. R. Schucany,et al.  Correlation structure in Farlie-Gumbel-Morgenstern distributions , 1978 .

[40]  L. Mead,et al.  Maximum entropy in the problem of moments , 1984 .

[41]  P. Bickel,et al.  Efficient estimation of linear functionals of a probability measure P with known marginal distributions , 1991 .

[42]  S. Kullback.  Probability Densities with Given Marginals , 1968 .

[43]  Alan J. Lee,et al.  Generating Random Binary Deviates Having Fixed Marginal Distributions and Specified Degrees of Association , 1993 .

[44]  H. Ryu.  Maximum entropy estimation of density and regression functions , 1993 .

[45]  George G. Judge,et al.  Econometric foundations , 2000 .

[46]  M. Sklar.  Fonctions de repartition a n dimensions et leurs marges , 1959 .

[47]  P. Hall.  On Kullback-Leibler loss and density estimation , 1987 .

[48]  E. Käärik,et al.  Generation and Investigation of Multivariate Distributions having Fixed Discrete Marginals , 1996 .

[49]  Paul L. Fackler,et al.  Modeling Interdependence: An Approach to Simulation and Elicitation , 1991 .

[50]  H. Joe.  Multivariate models and dependence concepts , 1998 .

[51]  Emiliano A. Valdez,et al.  Annuity Valuation with Dependent Mortality , 1996 .

[52]  Nader Ebrahimi,et al.  Testing exponentiality based on Kullback-Leibler information , 1992 .

[53]  Jeffrey S. Racine,et al.  Entropy and predictability of stock market returns , 2002 .

[54]  E. Maasoumi.  A compendium to information theory in economics and econometrics , 1993 .

[55]  R. Nelsen.  An Introduction to Copulas , 1998 .

[56]  A. Barron,et al.  APPROXIMATION OF DENSITY FUNCTIONS BY SEQUENCES OF EXPONENTIAL FAMILIES , 1991 .

[57]  Josef Štěpán,et al.  Distributions with given marginals and moment problems , 1997 .

[58]  Harry Joe,et al.  Parametric families of multivariate distributions with given margins , 1993 .

[59]  H. Joe.  Relative Entropy Measures of Multivariate Dependence , 1989 .