Likelihood Ratio Exponential Families

The exponential family is well known in machine learning and statistical physics as the maximum entropy distribution subject to a set of observed constraints [1], while the geometric mixture path is common in MCMC methods such as annealed importance sampling (AIS) [2, 3]. Linking these two ideas, recent work [4] has interpreted the geometric mixture path as an exponential family of distributions to analyse the thermodynamic variational objective (TVO) [5]. We extend likelihood ratio exponential families to include solutions to ratedistortion (RD) optimization [6, 7], the Information Bottleneck (IB) method [8], and recent rate-distortion-classification (RDC) approaches which combine RD and IB [9, 10]. This provides a common mathematical framework for understanding these methods via the conjugate duality of exponential families and hypothesis testing. Further, we collect existing results [11–14] to provide a variational representation of intermediate RD or TVO distributions as a minimizing an expectation of KL divergences. This solution also corresponds to a size-power tradeoff using the likelihood ratio test and the Neyman Pearson lemma. In thermodynamic integration (TI) bounds [15, 16] such as the TVO, we identify the intermediate distribution whose expected sufficient statistics match the log partition function.