Low complexity, high performance algorithms for estimation and decoding

The tremendous success of the graphical error-correcting codes has generated great interest in iterative message-passing algorithms for approximate decoding and estimation; indeed the applications of such approximation techniques extend to a surprisingly vast range of fields of science and engineering, from thermal physics to artificial intelligence, and from computer vision to communications and information theory. In this thesis we will discuss two novel approaches to the general estimation problem based on graphs; each approach results in methods which are superior, in terms of performance and/or complexity, to the existing algorithms that solve the same problem exactly or approximately. In the first approach we reformulate the general estimation problem in a measure-theoretic framework, free of the notion of ‘variables’, upon which the conventional ‘junction tree’ based methods heavily rely. This enables us to exploit all the underlying structure in the data of the problem, in order to reduce the complexity of the underlying marginalization task. We introduce appropriate notions of independence and junction trees, and the corresponding measure-theoretic junction tree algorithm, and we give an automatic procedure, called ‘lifting’, that finds such low complexity junction trees for a given marginalization problem. This automatic procedure often gives rise to algorithms that are substantially less complex than those obtained using the conventional methods. The second approach is based on a recently discovered connection between the fixed points of the well-known loopy belief propagation algorithm, and certain ‘variational free energy’ concepts from statistical physics. We discuss the general ‘Kikuchi approximation method’ for solving a marginalization problem. We define appropriate concepts of graphs and minimal graphs representing such problems, and derive several results on the strengths and limitations of the Kikuchi method based on the graph-theoretic properties of these graphs. In addition to theoretical discussions, for each approach we provide detailed examples and simulation results on several applications of interest, such as exact decoding of low-density parity-check (LDPC) codes, and joint decoding of a partial response magnetic recording channel and an LDPC code.