Information Theory Preliminaries

1.1 Scope The quality of any decision depends both upon the amount of relevant information available as well as the quality of the decision-making process itself. In this book, we investigate decision making in the context of detection and estimation problems that are statistically modeled according to some assumed probability distribution. Assumptions become necessary to help find true solutions when facing problems where available information is insufficient. Appropriate assumptions provide guidance when selecting the best solution from a given set of candidates. However, new assumptions impart new information. Consequently, making the minimum number of assumptions consistent with available evidence is vital; making these assumptions using principled methods is also key. Given the important role of information in these problems, we appeal to information theory as a principled approach to making appropriate assumptions and assigning meaningful models. Much of this book deals with the application of information-theoretic notions to problems of estimation. Following a brief review of information theory basics, we introduce the inverse problem and its role in signal restoration. Next, we review both classical and recent methods for solving the inverse problem. Part I of this monograph explores density-estimation techniques. We first review one such technique-maximum-entropy (ME) estimation. Then, we present a novel result which establishes confidence bounds on these estimates. In Part II, we introduce a framework for solving inverse problems to find unique solutions that maintain fidelity with observed data. We conclude the book with some examples and applications of the proposed algorithms to common problems encountered in signal processing.