On optimum strategies for minimizing the exponential moments of a loss function

We consider a general problem of minimizing the exponential moment of a given loss function, with an emphasis on the relation to the more common criterion of minimization the first moment of the same loss function. Our basic observation is about simple sufficient conditions for a strategy to be optimum in the exponential moment sense. This observation is useful and application examples are given. We also examine the asymptotic regime and investigate universal asymptotically optimum strategies in light of the aforementioned sufficient conditions.

[1]  Ehud Weinstein,et al.  A lower bound on the mean-square error in random parameter estimation , 1985, IEEE Trans. Inf. Theory.

[2]  Lee D. Davisson,et al.  The prediction error of stationary Gaussian time series of unknown covariance , 1965, IEEE Trans. Inf. Theory.

[3]  B. Derrida Random-energy model: An exactly solvable model of disordered systems , 1981 .

[4]  Robert J. Elliott,et al.  Risk-sensitive generalizations of minimum variance estimation and control , 1997 .

[5]  K. H. Barratt Digital Coding of Waveforms , 1985 .

[6]  Neri Merhav On Optimum Parameter Modulation–Estimation From a Large Deviations Perspective , 2012, IEEE Transactions on Information Theory.

[7]  LARGE DEVIATIONS AND BERRY-ESSEEN IN- EQUALITIES FOR ESTIMATORS IN NONLINEAR NONHOMOGENEOUS DIFFUSIONS , 2007 .

[8]  P. Whittle Risk-sensitive linear/quadratic/gaussian control , 1981, Advances in Applied Probability.

[9]  Neri Merhav,et al.  The Shannon cipher system with a guessing wiretapper , 1999, IEEE Trans. Inf. Theory.

[10]  R. Gray,et al.  Vector quantization , 1984, IEEE ASSP Magazine.

[11]  James A. Bucklew,et al.  Companding and random quantization in several dimensions , 1981, IEEE Trans. Inf. Theory.

[12]  Te Sun Han,et al.  The optimal overflow and underflow probabilities with variable-length coding for the general source , 2000, 2000 IEEE International Symposium on Information Theory (Cat. No.00CH37060).

[13]  A.D. Wyner,et al.  Fundamental limits in information theory , 1981, Proceedings of the IEEE.

[14]  J. Massey Guessing and entropy , 1994, Proceedings of 1994 IEEE International Symposium on Information Theory.

[15]  M. Mézard,et al.  Information, Physics, and Computation , 2009 .

[16]  Neri Merhav,et al.  Joint Source-Channel Coding and Guessing with Application to Sequential Decoding , 1998, IEEE Trans. Inf. Theory.

[17]  G. Longo Source Coding Theory , 1970 .

[18]  Robert J. Elliott,et al.  Risk Sensitive Generalization of Minimum Variance Estimation and Control , 1995 .

[19]  Neri Merhav,et al.  Guessing Subject to Distortion , 1998, IEEE Trans. Inf. Theory.

[20]  Lukasz Stettner,et al.  Risk-Sensitive Control of Discrete-Time Markov Processes with Infinite Horizon , 1999, SIAM J. Control. Optim..

[21]  Seymour Sherman,et al.  Non-mean-square error criteria , 1958, IRE Trans. Inf. Theory.

[22]  A. Kester,et al.  Large Deviations of Estimators , 1986 .

[23]  Nariman Farvardin,et al.  On overflow and underflow problems in buffer-instrumented variable-length coding of fixed-rate memoryless sources , 1986, IEEE Trans. Inf. Theory.

[24]  Bernd Girod,et al.  Quantization effects on digital watermarks , 2001, Signal Process..

[25]  Neri Merhav,et al.  Source coding exponents for zero-delay coding with finite memory , 2003, IEEE Trans. Inf. Theory.

[26]  Robert M. Gray,et al.  An Algorithm for Vector Quantizer Design , 1980, IEEE Trans. Commun..

[27]  Wolfgang J. Runggaldier,et al.  Connections between stochastic control and dynamic games , 1996, Math. Control. Signals Syst..

[28]  Neri Merhav The Generalized Random Energy Model and its Application to the Statistical Physics of Ensembles of Hierarchical Codes , 2009, IEEE Transactions on Information Theory.

[29]  A. D. Wyner Another look at the coding theorem of information theory—A tutorial , 1970 .

[30]  Erdal Arikan An inequality on guessing and its application to sequential decoding , 1996, IEEE Trans. Inf. Theory.

[31]  B. Derrida Random-Energy Model: Limit of a Family of Disordered Models , 1980 .

[32]  Frederick Jelinek,et al.  Buffer overflow in variable length coding of fixed rate sources , 1968, IEEE Trans. Inf. Theory.

[33]  丸山 徹 Convex Analysisの二,三の進展について , 1977 .

[34]  J. Lynch,et al.  A weak convergence approach to the theory of large deviations , 1997 .

[35]  Ian R. Petersen,et al.  Robust Properties of Risk-Sensitive Control , 2000, Math. Control. Signals Syst..

[36]  I. M. Jacobs,et al.  Principles of Communication Engineering , 1965 .

[37]  Neri Merhav,et al.  A strong version of the redundancy-capacity theorem of universal coding , 1995, IEEE Trans. Inf. Theory.

[38]  Dimitri P. Bertsekas,et al.  Dynamic Programming and Optimal Control, Two Volume Set , 1995 .

[39]  B.D. Steinberg,et al.  Principles of communication , 1978, Proceedings of the IEEE.

[40]  P. Whittle A risk-sensitive maximum principle , 1990 .

[41]  A. Wyner On the Probability of Buffer Overflow Under an Arbitrary Bounded Input-Output Distribution , 1974 .

[42]  Harry L. Van Trees,et al.  Detection, Estimation, and Modulation Theory, Part I , 1968 .

[43]  W. Fleming,et al.  Risk sensitive control of finite state machines on an infinite horizon. I , 1997, Proceedings of the 36th IEEE Conference on Decision and Control.

[44]  Bernard Derrida,et al.  The random energy model , 1980 .

[45]  Katalin Marton,et al.  Error exponent for source coding with a fidelity criterion , 1974, IEEE Trans. Inf. Theory.

[46]  R. Howard,et al.  Risk-Sensitive Markov Decision Processes , 1972 .

[47]  Norman Margolus,et al.  Physics and Computation , 1987 .

[48]  Imre Csiszár On the error exponent of source-channel transmission with a distortion threshold , 1982, IEEE Trans. Inf. Theory.

[49]  Neri Merhav,et al.  Universal coding with minimum probability of codeword length overflow , 1991, IEEE Trans. Inf. Theory.

[50]  James A. Bucklew A note on optimal multidimensional companders , 1983, IEEE Trans. Inf. Theory.

[51]  David L. Neuhoff,et al.  Quantization , 2022, IEEE Trans. Inf. Theory.

[52]  A. Wyner,et al.  On communication of analog data from a bounded source space , 1969 .

[53]  Paul Dupuis,et al.  An Escape-Time Criterion for Queueing Networks: Asymptotic Risk-Sensitive Control via Differential Games , 2003, Math. Oper. Res..

[54]  Pierre A. Humblet Generalization of Huffman coding to minimize the probability of buffer overflow , 1981, IEEE Trans. Inf. Theory.

[55]  Neri Merhav,et al.  Hierarchical Guessing with a Fidelity Criterion , 1999, IEEE Trans. Inf. Theory.

[56]  LARGE DEVIATION PROBABILITIES FOR MAXIMUM LIKELIHOOD ESTIMATOR AND BAYES ESTIMATOR OF A PARAMETER FOR FRACTIONAL ORNSTEIN-UHLENBECK TYPE PROCESS , 2006 .

[57]  Thomas M. Cover,et al.  Elements of Information Theory: Cover/Elements of Information Theory, Second Edition , 2005 .

[58]  Neri Merhav,et al.  Universal Prediction , 1998, IEEE Trans. Inf. Theory.

[59]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[60]  Allen Gersho,et al.  Principles of quantization , 1978 .

[61]  Gregory W. Wornell,et al.  Quantization index modulation: A class of provably good methods for digital watermarking and information embedding , 2001, IEEE Trans. Inf. Theory.

[62]  David L. Neuhoff,et al.  Causal source codes , 1982, IEEE Trans. Inf. Theory.

[63]  David L. Neuhoff,et al.  Variable-to-fixed length codes provide better large deviations performance than fixed-to-variable length codes , 1992, IEEE Trans. Inf. Theory.

[64]  Jorma Rissanen,et al.  Universal coding, information, prediction, and estimation , 1984, IEEE Trans. Inf. Theory.

[65]  Toby Berger,et al.  Rate distortion theory : a mathematical basis for data compression , 1971 .

[66]  Neri Merhav On optimum strategies for minimizing the exponential moments of a given cost function , 2011, ArXiv.

[67]  Daniel R. Fuhrmann,et al.  Improving the visual quality of JPEG-encoded images via companding , 1997, J. Electronic Imaging.