Information and Estimation Theory

Shannon’s seminal paper of 1948 [32] is popularly seen as the cornerstone of information theory. Arguably Shannon’s paper addressed a significant problem of the age, namely the scientific quantification of the rather abstract concept of information. Despite numerous earlier works, predominantly by Nyquist, Kupfmuller, Gabor, and Hartley [15, 17, 22, 28], each fundamentally failed in a number of ways. Foremost of these deficiencies is that no rigorous treatment was given to the role of noise, although its importance was noted by both Nyquist and Hartley. Noise is an essential component to information transfer because random, and hence unpredictable, perturbations to an information carrying signal ultimately prohibit exact determination of the original signal or its implicit meaning. The probabilistic treatment of Shannon (and later contributors [30, 34, 39]) allowed the stochastic nature of noise to be considered in a concise and rigorous fashion, and thus a definition for information was born.