Universal Simulation With Fidelity Criteria

We consider the problem of universal simulation of a memoryless source (with some partial extensions to Markov sources), based on a training sequence emitted from the source. The objective is to maximize the conditional entropy of the simulated sequence given the training sequence, subject to a certain distance constraint between the probability distribution of the output sequence and the probability distribution of the input, training sequence. We derive, for several distance criteria, single-letter expressions for the maximum attainable conditional entropy as well as corresponding universal simulation schemes that asymptotically attain these maxima.

[1]  Andrew R. Barron,et al.  Information-theoretic asymptotics of Bayes methods , 1990, IEEE Trans. Inf. Theory.

[2]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[3]  Sanjeev R. Kulkarni,et al.  Separation of random number generation and resolvability , 2000, IEEE Trans. Inf. Theory.

[4]  Neri Merhav.  Achievable key rates for universal simulation of random data with respect to a set of statistical tests , 2004, IEEE Transactions on Information Theory.

[5]  Peter J. Huber,et al.  Robust Statistical Procedures: Second Edition , 1996 .

[6]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[7]  Gadiel Seroussi,et al.  On universal types , 2004, IEEE Transactions on Information Theory.

[8]  Sergio Verdú,et al.  Simulation of random processes and rate-distortion theory , 1996, IEEE Trans. Inf. Theory.

[9]  P. J. Huber.  Robust Statistical Procedures , 1977 .

[10]  Mamoru Hoshi,et al.  Interval algorithm for random number generation , 1997, IEEE Trans. Inf. Theory.

[11]  Neri Merhav,et al.  On universal simulation of information sources using training data , 2004, IEEE Transactions on Information Theory.

[12]  Erik Ordentlich,et al.  Universal portfolios with side information , 1996, IEEE Trans. Inf. Theory.

[13]  Aarnout Brombacher,et al.  Probability... , 2009, Qual. Reliab. Eng. Int..

[14]  Neri Merhav,et al.  Universal Delay-Limited Simulation , 2005, IEEE Transactions on Information Theory.

[15]  Sergio Verdú,et al.  Channel simulation and coding with side information , 1994, IEEE Trans. Inf. Theory.

[16]  Sergio Verdú,et al.  Approximation theory of output statistics , 1993, IEEE Trans. Inf. Theory.

[17]  Robert M. Gray,et al.  Time-invariant trellis encoding of ergodic discrete-time sources with a fidelity criterion , 1977, IEEE Trans. Inf. Theory.

[18]  Robert M. Gray,et al.  Probability, Random Processes, And Ergodic Properties , 1987 .

[19]  N. Merhav,et al.  Addendum to "On Universal Simulation of Information Sources Using Training Data , 2005, IEEE Trans. Inf. Theory.

[20]  S.A. Kassam,et al.  Robust techniques for signal processing: A survey , 1985, Proceedings of the IEEE.