Maximum Entropy, Likelihood and Uncertainty: A Comparison

A framework for comparing the maximum likelihood (ML) and maximum entropy (ME) approaches is developed. Two types of linear models are considered. In the first type, the objective is to estimate probability distributions given some moment conditions. In this case the ME and ML are equivalent. A generalization of this type of estimation models to incorporate noisy data is discussed as well. The second type of models encompasses the traditional linear regression type models where the number of observations is larger than the number of unknowns and the objects to be inferred are not natural probabilities. After reviewing a generalized ME estimator and the empirical likelihood (or weighted least squares) estimator, the two are compared and contrasted with the ML. It is shown that, in general, the ME estimators use less input information and may be viewed, within the second type models, as expected log-likelihood estimators. In terms of informational ranking, if the objective is to estimate with minimum a-priori assumptions, then the generalized ME estimator is superior to the other estimators. Two detailed examples, reflecting the two types of models, are discussed. The first example deals with estimating a first order Markov process. In the second example the empirical (natural) weights of each observation, together with the other unknowns, are the subject of interest.

[1]  R. Levine,et al.  An information theoretical approach to inversion problems , 1980 .

[2]  E. Jaynes.  Information Theory and Statistical Mechanics , 1957 .

[3]  Art B. Owen,et al.  Empirical Likelihood for Linear Models , 1991 .

[4]  Arnold Zellner.  Bayesian Method of Moments (BMOM) Analysis of Mean and Regression Models , 1996 .

[5]  Peter Hall,et al.  Pseudo-Likelihood Theory for Empirical Likelihood , 1990 .

[6]  T. Bewley,et al.  Advances in Econometrics , 1995 .

[7]  Rodney W. Johnson,et al.  Axiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy , 1980, IEEE Trans. Inf. Theory.

[8]  C. Ray Smith,et al.  Maximum-entropy and Bayesian methods in science and engineering , 1988 .

[9]  I. Csiszár.  Why least squares and maximum entropy? An axiomatic approach to inference for linear inverse problems , 1991 .

[10]  A. Owen.  Empirical Likelihood Ratio Confidence Regions , 1990 .

[11]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[12]  G. Imbens,et al.  Imposing Moment Restrictions by Weighting , 1999 .

[13]  J. Bernardo.  Expected Information as Expected Utility , 1979 .

[14]  D. McFadden.  MEASUREMENT OF URBAN TRAVEL DEMAND , 1974 .

[15]  Douglas J. Miller,et al.  Maximum entropy econometrics: robust estimation with limited data , 1996 .

[16]  Solomon Kullback,et al.  Information Theory and Statistics , 1960 .

[17]  J. Lawless,et al.  Empirical Likelihood and General Estimating Equations , 1994 .

[18]  R. Levine,et al.  An Algorithm for Finding the Distribution of Maximal Entropy , 1979 .

[19]  Thomas J. DiCiccio,et al.  Empirical Likelihood is Bartlett-Correctable , 1991 .

[20]  J. Skilling.  The Axioms of Maximum Entropy , 1988 .

[21]  I. Good.  Maximum Entropy for Hypothesis Formulation, Especially for Multidimensional Contingency Tables , 1963 .

[22]  A. Zellner.  Optimal Information Processing and Bayes's Theorem , 1988 .