Information Measures for Discrete Systems

In this chapter, we define Shannon’s information measures for discrete-time discrete-alphabet systems from a probabilistic standpoint and develop their properties. Elucidating the operational significance of probabilistically defined information measures vis-a-vis the fundamental limits of coding constitutes a main objective of this book; this will be seen in the subsequent chapters.