Lower Bounds on Exponential Moments of the Quadratic Error in Parameter Estimation

Considering the problem of risk-sensitive parameter estimation, we propose a fairly wide family of lower bounds on the exponential moments of the quadratic error, both in the Bayesian and the non-Bayesian regime. This family of bounds, which is based on a change of measures, offers considerable freedom in the choice of the reference measure, and our efforts are devoted to explore this freedom to a certain extent. Our focus is mostly on signal models that are relevant to communication problems, namely, models of a parameter-dependent signal (modulated signal) corrupted by additive white Gaussian noise, but the methodology proposed is also applicable to other types of parametric families, such as models of linear systems driven by random input signals (white noise, in most cases), and others. In addition to the well-known motivations of the risk-sensitive cost function (i.e., the exponential quadratic cost function), which is most notably, the robustness to model uncertainty, we also view this cost function as a tool for studying fundamental limits concerning the tail behavior of the estimation error. Another interesting aspect that we demonstrate in a certain parametric model is that the risk-sensitive cost function may be subjected to phase transitions, owing to some analogies with statistical mechanics.

[1]  R. Howard,et al.  Risk-Sensitive Markov Decision Processes , 1972 .

[2]  Paul Dupuis,et al.  Robust Bounds on Risk-Sensitive Functionals via Rényi Divergence , 2013, SIAM/ASA J. Uncertain. Quantification.

[3]  S.,et al.  Risk-Sensitive Control and Dynamic Games for Partially Observed Discrete-Time Nonlinear Systems , 1994 .

[4]  R. Gallager Information Theory and Reliable Communication , 1968 .

[5]  Ravi N. Banavar,et al.  Properties of risk-sensitive filters/estimators , 1998 .

[6]  Neri Merhav On optimum strategies for minimizing the exponential moments of a loss function , 2011, Commun. Inf. Syst..

[7]  J. Kiefer On Minimum Variance Estimators , 1952 .

[8]  Steven I. Marcus,et al.  Estimation of hidden markov models: risk-sensitive filter banks and qualitative analysis of their sample paths , 2002, IEEE Trans. Autom. Control..

[9]  Lawrence D. Brown,et al.  Bounds on the Bayes and minimax risk for signal parameter estimation , 1993, IEEE Trans. Inf. Theory.

[10]  J. Lynch,et al.  A weak convergence approach to the theory of large deviations , 1997 .

[11]  W. Fleming,et al.  Risk-Sensitive Control of Finite State Machines on an Infinite Horizon I , 1997 .

[12]  Peng Cui,et al.  Risk-sensitive fixed-point smoothing estimation for linear discrete-time systems with multiple output delays , 2013, J. Syst. Sci. Complex..

[13]  Kristine L. Bell,et al.  Bayesian Bounds for Parameter Estimation and Nonlinear Filtering/Tracking , 2007 .

[14]  Robert J. Elliott,et al.  Risk-sensitive generalizations of minimum variance estimation and control , 1997 .

[15]  P. Whittle A risk-sensitive maximum principle , 1990 .

[16]  P. Dupuis,et al.  Robust Properties of Risk-Sensitive Control , 1998, Proceedings of the 37th IEEE Conference on Decision and Control (Cat. No.98CH36171).

[17]  P. Whittle Risk-sensitive linear/quadratic/gaussian control , 1981, Advances in Applied Probability.

[18]  L. Brown,et al.  Ensemble Minimax Estimation for Multivariate Normal Means , 2011 .

[19]  Irwin Guttman,et al.  Bhattacharyya Bounds without Regularity Assumptions , 1952 .

[20]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[21]  S. Stenholm Information, Physics and Computation, by Marc Mézard and Andrea Montanari , 2010 .

[22]  M. Srinivasan,et al.  Robust Fading Channel Estimation under Parameter and Process Noise Uncertainty with Risk Sensitive Filter and Its comparison with CRLB , 2014 .

[23]  Jason J. Ford Risk-sensitive filtering and parameter estimation , 1999 .

[24]  Neri Merhav,et al.  Information-Theoretic Applications of the Logarithmic Probability Comparison Bound , 2015, IEEE Trans. Inf. Theory.

[25]  L. Brown,et al.  Information Inequality Bounds on the Minimax Risk (with an Application to Nonparametric Regression) , 1991 .

[26]  Neri Merhav,et al.  Statistical Physics and Information Theory , 2010, Found. Trends Commun. Inf. Theory.

[27]  Wolfgang J. Runggaldier,et al.  Connections between stochastic control and dynamic games , 1996, Math. Control. Signals Syst..

[28]  Lukasz Stettner,et al.  Risk-Sensitive Control of Discrete-Time Markov Processes with Infinite Horizon , 1999, SIAM J. Control. Optim..

[29]  D. G. Chapman,et al.  Minimum Variance Estimation Without Regularity Assumptions , 1951 .

[30]  E. Barankin Locally Best Unbiased Estimates , 1949 .