On Zero-Delay Source-Channel Coding

This paper studies the zero-delay source-channel coding problem, and specifically the problem of obtaining the vector transformations that optimally map between the m-dimensional source space and k-dimensional channel space, under a given transmission power constraint and for the mean square error distortion. The functional properties of the cost are studied and the necessary conditions for the optimality of the encoder and decoder mappings are derived. An optimization algorithm that imposes these conditions iteratively, in conjunction with the noisy channel relaxation method to mitigate poor local minima, is proposed. The numerical results show strict improvement over prior methods. The numerical approach is extended to the scenario of source-channel coding with decoder side information. The resulting encoding mappings are shown to be continuous relatives of, and in fact subsume as special case, the Wyner-Ziv mappings encountered in digital distributed source coding systems. A well-known result in information theory pertains to the linearity of optimal encoding and decoding mappings in the scalar Gaussian source and channel setting, at all channel signal-to-noise ratios (CSNRs). In this paper, the linearity of optimal coding, beyond the Gaussian source and channel, is considered and the necessary and sufficient condition for linearity of optimal mappings, given a noise (or source) distribution, and a specified a total power constraint are derived. It is shown that the Gaussian source-channel pair is unique in the sense that it is the only source-channel pair for which the optimal mappings are linear at more than one CSNR values. Moreover, the asymptotic linearity of optimal mappings is shown for low CSNR if the channel is Gaussian regardless of the source and, at the other extreme, for high CSNR if the source is Gaussian, regardless of the channel. The extension to the vector settings is also considered where besides the conditions inherited from the scalar case, additional constraints must be satisfied to ensure linearity of the optimal mappings.

[1]  Ram Zamir,et al.  Bounds for joint source-channel coding at high SNR , 2011, 2011 IEEE International Symposium on Information Theory Proceedings.

[2]  Michael Gastpar,et al.  Reduced-Dimension Linear Transform Coding of Correlated Signals in Networks , 2012, IEEE Transactions on Signal Processing.

[3]  Kenneth Rose,et al.  On Conditions for Linearity of Optimal Estimation , 2010, IEEE Transactions on Information Theory.

[4]  丸山 徹.  Convex Analysisの二,三の進展について , 1977 .

[5]  Dudley,et al.  Real Analysis and Probability: Measurability: Borel Isomorphism and Analytic Sets , 2002 .

[6]  Sueli I. Rodrigues Costa,et al.  Curves on a sphere, shift-map dynamics, and error control for continuous alphabet sources , 2003, IEEE Transactions on Information Theory.

[7]  H. Witsenhausen.  A Counterexample in Stochastic Optimum Control , 1968 .

[8]  Yichuan Hu,et al.  Analog Joint Source-Channel Coding Using Non-Linear Curves and MMSE Decoding , 2011, IEEE Transactions on Communications.

[9]  Kyong-Hwa Lee,et al.  Optimal Linear Coding for Vector Channels , 1976, IEEE Trans. Commun..

[10]  Sergio Verdú,et al.  Functional Properties of Minimum Mean-Square Error and Mutual Information , 2012, IEEE Transactions on Information Theory.

[11]  Michael Gastpar,et al.  To code, or not to code: lossy source-channel communication revisited , 2003, IEEE Trans. Inf. Theory.

[12]  Cristian S. Calude.  The mathematical theory of information , 2007 .

[13]  C. Villani.  Optimal Transport: Old and New , 2008 .

[14]  Kenneth Rose,et al.  A deterministic annealing approach to Witsenhausen's counterexample , 2014, 2014 IEEE International Symposium on Information Theory.

[15]  Sang Joon Kim,et al.  A Mathematical Theory of Communication , 2006 .

[16]  Demosthenis Teneketzis,et al.  On the Structure of Optimal Real-Time Encoders and Decoders in Noisy Communication , 2006, IEEE Transactions on Information Theory.

[17]  Christos Papadimitriou,et al.  Intractable problems in control theory , 1985, 1985 24th IEEE Conference on Decision and Control.

[18]  R. M. Dudley,et al.  Real Analysis and Probability , 1989 .

[19]  Kenneth Rose,et al.  Robust vector quantizer design by noisy channel relaxation , 1999, IEEE Trans. Commun..

[20]  Meir Feder,et al.  Distortion lower bounds for finite dimensional joint source-channel coding , 2008, 2008 IEEE International Symposium on Information Theory.

[21]  Mikael Skoglund,et al.  Polynomial based analog source-channel codes , 2009 .

[22]  Ertem Tuncel,et al.  Zero-Delay Joint Source-Channel Coding Using Hybrid Digital-Analog Schemes in the Wyner-Ziv Setting , 2014, IEEE Transactions on Communications.

[23]  Kenneth Rose,et al.  On Optimal Jamming Over an Additive Noise Channel , 2013, 52nd IEEE Conference on Decision and Control.

[24]  Thomas J. Goblick,et al.  Theoretical limitations on the transmission of data from analog sources , 1965, IEEE Trans. Inf. Theory.

[25]  Kenneth Rose,et al.  On linear transforms in zero-delay Gaussian source channel coding , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[26]  Jacob Ziv,et al.  The behavior of analog communication systems , 1970, IEEE Trans. Inf. Theory.

[27]  Tor A. Ramstad,et al.  Using 2:1 Shannon mapping for joint source-channel coding , 2005, Data Compression Conference.

[28]  Serdar Yüksel,et al.  On Optimal Causal Coding of Partially Observed Markov Sources in Single and Multiterminal Settings , 2010, IEEE Transactions on Information Theory.

[29]  Imre Csiszár,et al.  Information Theory - Coding Theorems for Discrete Memoryless Systems, Second Edition , 2011 .

[30]  Neri Merhav,et al.  Data processing lower bounds for scalar lossy source codes with side information at the decoder , 2012, 2012 IEEE International Symposium on Information Theory Proceedings.

[31]  Sae-Young Chung,et al.  On the construction of some capacity-approaching coding schemes , 2000 .

[32]  Ertem Tuncel,et al.  Zero-delay joint source-channel coding for the Gaussian Wyner-Ziv problem , 2011, 2011 IEEE International Symposium on Information Theory Proceedings.

[33]  Bülent Sankur,et al.  Performance bounds and optimal linear coding for discrete-time multichannel communication systems (Corresp.) , 1980, IEEE Trans. Inf. Theory.

[34]  Kenneth Rose,et al.  A deterministic annealing approach to optimization of zero-delay source-channel codes , 2013, 2013 IEEE Information Theory Workshop (ITW).

[35]  Kenneth Rose,et al.  Optimization of zero-delay mappings for distributed coding by deterministic annealing , 2014, 2014 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[36]  H. Witsenhausen.  On the structure of real-time source coders , 1979, The Bell System Technical Journal.

[37]  C.E. Shannon,et al.  Communication in the Presence of Noise , 1949, Proceedings of the IRE.

[38]  Mikael Skoglund,et al.  Optimized low-delay source-channel-relay mappings , 2010, IEEE Transactions on Communications.

[39]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[40]  T. Başar.  A trace minimization problem with applications in joint estimation and control under nonclassical information , 1980 .

[41]  Terrence L. Fine,et al.  Properties of an optimum digital system and applications , 1964, IEEE Trans. Inf. Theory.

[42]  Jack K. Wolf,et al.  Noiseless coding of correlated information sources , 1973, IEEE Trans. Inf. Theory.

[43]  Thomas M. Cover,et al.  Elements of Information Theory , 2005 .

[44]  D. Luenberger.  Optimization by Vector Space Methods , 1968 .

[45]  Mikael Skoglund,et al.  Hybrid Digital–Analog Source–Channel Coding for Bandwidth Compression/Expansion , 2006, IEEE Transactions on Information Theory.

[46]  J. Norris.  Appendix: probability and measure , 1997 .

[47]  Kenneth Rose,et al.  On zero delay source-channel coding: Functional properties and linearity conditions , 2013, 2013 IEEE International Symposium on Information Theory.

[48]  Jan Khre,et al.  The Mathematical Theory of Information , 2012 .

[49]  Mikael Skoglund,et al.  Polynomial based analog source-channel codes - [transactions papers] , 2009, IEEE Transactions on Communications.

[50]  Tamer Basar,et al.  With the Capacity 0.461(bits) and the Optimal Opd Being 'q = , 1998 .

[51]  George L. Turin,et al.  The theory of optimum noise immunity , 1959 .

[52]  Tsachy Weissman,et al.  Universal Zero-Delay Joint Source–Channel Coding , 2006, IEEE Transactions on Information Theory.

[53]  Tamás Linder,et al.  Optimization and convergence of observation channels in stochastic control , 2010, Proceedings of the 2011 American Control Conference.

[54]  Tamer Basar,et al.  Variations on the theme of the Witsenhausen counterexample , 2008, 2008 47th IEEE Conference on Decision and Control.

[55]  Tor A. Ramstad,et al.  Bandwidth compression for continuous amplitude channels based on vector approximation to a continuous subset of the source signal space , 1997, 1997 IEEE International Conference on Acoustics, Speech, and Signal Processing.

[56]  Jean C. Walrand,et al.  Optimal causal coding - decoding problems , 1983, IEEE Trans. Inf. Theory.

[57]  Aaron D. Wyner,et al.  The rate-distortion function for source coding with side information at the decoder , 1976, IEEE Trans. Inf. Theory.

[58]  Neri Merhav,et al.  Data-Processing Bounds for Scalar Lossy Source Codes With Side Information at the Decoder , 2013, IEEE Transactions on Information Theory.

[59]  T.A. Ramstad,et al.  Power Constrained Channel Optimized Vector Quantizers used for Bandwidth Expansion , 2007, 2007 4th International Symposium on Wireless Communication Systems.

[60]  Sergio Verdú,et al.  Witsenhausen's counterexample: A view from optimal transport theory , 2011, IEEE Conference on Decision and Control and European Control Conference.

[61]  Tor A. Ramstad,et al.  Shannon-kotel-nikov mappings in joint source-channel coding , 2009, IEEE Transactions on Communications.

[62]  Nam C. Phamdo,et al.  Hybrid digital-analog (HDA) joint source-channel codes for broadcasting and robust communications , 2002, IEEE Trans. Inf. Theory.

[63]  K. Rose.  Deterministic annealing for clustering, compression, classification, regression, and related optimization problems , 1998, Proc. IEEE.

[64]  Kenneth Rose,et al.  Optimized Analog Mappings for Distributed Source-Channel Coding , 2010, 2010 Data Compression Conference.

[65]  Kenneth Rose,et al.  Optimal mappings for joint source channel coding , 2010, 2010 IEEE Information Theory Workshop on Information Theory (ITW 2010, Cairo).