Extension of the Blahut-Arimoto algorithm for maximizing directed information

We extend the Blahut-Arimoto algorithm for maximizing Massey's directed information, which can be used for estimating the capacity of channels with delayed feedback. In order to do so, we apply the ideas from the regular Blahut-Arimoto algorithm, i.e., the alternating maximization procedure, onto our new problem, and show its convergence to the global optimum value. Our main insight in this paper is that in order to find the maximum of the directed information over causal conditioning probability mass function, one can use a backward index time maximization combined with the alternating maximization procedure. We give a detailed description of the algorithm, its complexity, and memory needed. Some numerical results are provided to illustrate the performance of the suggested algorithm.

[1]  Gerald Matz,et al.  Information geometric formulation and interpretation of accelerated Blahut-Arimoto-type algorithms , 2004, Information Theory Workshop.

[2]  Andrea J. Goldsmith,et al.  Finite State Channels With Time-Invariant Deterministic Feedback , 2006, IEEE Transactions on Information Theory.

[3]  J. Massey.  CAUSALITY, FEEDBACK AND DIRECTED INFORMATION , 1990 .

[4]  Toby Berger,et al.  Capacity and zero-error capacity of Ising channels , 1990, IEEE Trans. Inf. Theory.

[5]  R. Gallager.  Information Theory and Reliable Communication , 1968 .

[6]  Abbas El Gamal,et al.  On the capacity of computer memory with defects , 1983, IEEE Trans. Inf. Theory.

[7]  Ramji Venkataramanan,et al.  Source Coding With Feed-Forward: Rate-Distortion Theorems and Error Exponents for a General Source , 2007, IEEE Transactions on Information Theory.

[8]  Claude E. Shannon,et al.  The zero error capacity of a noisy channel , 1956, IRE Trans. Inf. Theory.

[9]  Garik Markarian,et al.  A modified Blahut algorithm for decoding Reed-Solomon codes beyond half the minimum distance , 2004, IEEE Transactions on Communications.

[10]  Young-Han Kim,et al.  Feedback Capacity of Stationary Gaussian Channels , 2006, 2006 IEEE International Symposium on Information Theory.

[11]  Haim H. Permuter,et al.  Capacity of the Trapdoor Channel With Feedback , 2006, IEEE Transactions on Information Theory.

[12]  Todd P. Coleman,et al.  On reversible Markov chains and maximization of directed information , 2010, 2010 IEEE International Symposium on Information Theory.

[13]  Wei Yu,et al.  Blahut-Arimoto algorithms for computing channel capacity and rate-distortion with side information , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..

[14]  Y.-H. Kim,et al.  A Coding Theorem for a Class of Stationary Channels with Feedback , 2007, 2007 IEEE International Symposium on Information Theory.

[15]  Richard E. Blahut,et al.  Computation of channel capacity and rate-distortion functions , 1972, IEEE Trans. Inf. Theory.

[16]  Yossef Steinberg,et al.  Information embedding with reversible stegotext , 2009, 2009 IEEE International Symposium on Information Theory.

[17]  Robert B. Ash,et al.  Information Theory , 2020, The SAGE International Encyclopedia of Mass Media and Society.

[18]  Raymond W. Yeung,et al.  Information Theory and Network Coding , 2008 .

[19]  Suguru Arimoto,et al.  An algorithm for computing the capacity of arbitrary discrete memoryless channels , 1972, IEEE Trans. Inf. Theory.

[20]  Sekhar Tatikonda,et al.  The Capacity of Channels With Feedback , 2006, IEEE Transactions on Information Theory.

[21]  Tsachy Weissman,et al.  On competitive prediction and its relation to rate-distortion theory , 2003, IEEE Trans. Inf. Theory.

[22]  Sekhar Tatikonda,et al.  Feedback capacity of finite-state machine channels , 2005, IEEE Transactions on Information Theory.

[23]  Stephen P. Boyd,et al.  Convex Optimization , 2004, Algorithms and Theory of Computation Handbook.

[24]  Gerhard Kramer,et al.  Directed information for channels with feedback , 1998 .

[25]  H. Marko,et al.  The Bidirectional Communication Theory - A Generalization of Information Theory , 1973, IEEE Transactions on Communications.

[26]  Ramji Venkataramanan,et al.  On Evaluating the Rate-Distortion Function of Sources with Feed-Forward and the Capacity of Channels with Feedback , 2007, 2007 IEEE International Symposium on Information Theory.

[27]  C. E. SHANNON,et al.  A mathematical theory of communication , 1948, MOCO.

[28]  Alex J. Grant,et al.  A generalization of Arimoto-Blahut algorithm , 2004, International Symposium onInformation Theory, 2004. ISIT 2004. Proceedings..