The Blahut-Arimoto Algorithms

For a discrete memoryless channel p(y|x), the capacity $$C = \mathop {\max }\limits_{r(x)} I(X;Y),$$ (10.1) where X and Y are respectively the input and the output of the generic channel and r(x) is the input distribution, characterizes the maximum asymptotically achievable rate at which information can be transmitted through the channel reliably. The expression for C in (10.1) is called a single-letter characterization because it depends only the transition matrix of the generic channel but not on the block length n of a code for the channel. When both the input alphabet X and the output alphabet Y are finite, the computation of C becomes a finite-dimensional maximization problem.