An Optimal Approximation Algorithm for Bayesian Inference

Abstract Approximating the inference probability Pr[ X = x | E = e ] in any sense, even for a single evidence node E , is NP-hard. This result holds for belief networks that are allowed to contain extreme conditional probabilities—that is, conditional probabilities arbitrarily close to 0. Nevertheless, all previous approximation algorithms have failed to approximate efficiently many inferences, even for belief networks without extreme conditional probabilities. We prove that we can approximate efficiently probabilistic inference in belief networks without extreme conditional probabilities. We construct a randomized approximation algorithm—the bounded-variance algorithm —that is a variant of the known likelihood-weighting algorithm. The bounded-variance algorithm is the first algorithm with provably fast inference approximation on all belief networks without extreme conditional probabilities. From the bounded-variance algorithm, we construct a deterministic approximation algorithm using current advances in the theory of pseudorandom generators. In contrast to the exponential worst-case behavior of all previous deterministic approximations, the deterministic bounded-variance algorithm approximates inference probabilities in worst-case time that is subexponential 2 (log n ) d , for some integer d that is a linear function of the depth of the belief network.

[1]  David J. Spiegelhalter,et al.  Local computations with probabilities on graphical structures and their application to expert systems , 1990 .

[2]  Bruce D'Ambrosio,et al.  Incremental Probabilistic Inference , 1993, UAI.

[3]  Judea Pearl,et al.  Evidential Reasoning Using Stochastic Simulation of Causal Models , 1987, Artif. Intell..

[4]  Malcolm Pradhan,et al.  Optimal Monte Carlo Estimation of Belief Network Inference , 1996, UAI.

[5]  Yun Peng,et al.  A Probabilistic Causal Model for Diagnostic Problem Solving Part I: Integrating Symbolic Causal Inference with Numeric Probabilistic Inference , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[6]  Gregory F. Cooper,et al.  The Computational Complexity of Probabilistic Inference Using Bayesian Belief Networks , 1990, Artif. Intell..

[7]  Solomon Eyal Shimony,et al.  A new algorithm for finding MAP assignments to belief networks , 1990, UAI.

[8]  L. N. Kanal,et al.  Uncertainty in Artificial Intelligence 5 , 1990 .

[9]  Ross D. Shachter,et al.  Simulation Approaches to General Probabilistic Inference on Belief Networks , 2013, UAI.

[10]  A. Hasman,et al.  Probabilistic reasoning in intelligent systems: Networks of plausible inference , 1991 .

[11]  Eugene Santos,et al.  Belief Updating by Enumerating High-Probability Independence-based Assignments , 1994, UAI.

[12]  Noam Nisan,et al.  Approximations of general independent distributions , 1992, STOC '92.

[13]  David Poole,et al.  Average-Case Analysis of a Search Algorithm for Estimating Prior and Posterior Probabilities in Bayesian Networks with Extreme Probabilities , 1993, IJCAI.

[14]  Robert M. Fung,et al.  Backward Simulation in Bayesian Networks , 1994, UAI.

[15]  Yun Peng,et al.  A Probabilistic Causal Model for Diagnostic Problem Solving Part II: Diagnostic Strategy , 1987, IEEE Transactions on Systems, Man, and Cybernetics.

[16]  Steffen L. Lauritzen,et al.  Bayesian updating in causal probabilistic networks by local computations , 1990 .

[17]  Noam Nisan,et al.  Pseudorandom bits for constant depth circuits , 1991, Comb..

[18]  Avi Wigderson,et al.  Deterministic approximate counting of depth-2 circuits , 1993, [1993] The 2nd Israel Symposium on Theory and Computing Systems.

[19]  R. Martin Chavez,et al.  Approximating Probabilistic Inference in Bayesian Belief Networks , 1993, IEEE Trans. Pattern Anal. Mach. Intell..

[20]  Gregory F. Cooper,et al.  A randomized approximation algorithm for probabilistic inference on bayesian belief networks , 1990, Networks.

[21]  Eric Horvitz,et al.  A Bayesian analysis of simulation algorithms for inference in belief networks , 1993, Networks.

[22]  Eric Horvitz,et al.  Bounded Conditioning: Flexible Inference for Decisions under Scarce Resources , 2013, UAI 1989.

[23]  Max Henrion,et al.  Efficient Search-Based Inference for noisy-OR Belief Networks: TopEpsilon , 1996, UAI.

[24]  Gregory F. Cooper,et al.  NESTOR: A Computer-Based Medical Diagnostic Aid That Integrates Causal and Probabilistic Knowledge. , 1984 .

[25]  Kuo-Chu Chang,et al.  Weighing and Integrating Evidence for Stochastic Simulation in Bayesian Networks , 2013, UAI.

[26]  Max Henrion,et al.  Search-Based Methods to Bound Diagnostic Probabilities in Very Large Belief Nets , 1991, UAI.

[27]  Michael Luby,et al.  Approximating Probabilistic Inference in Bayesian Belief Networks is NP-Hard , 1993, Artif. Intell..

[28]  A. Yao.  Separating the polynomial-time hierarchy by oracles , 1985 .

[29]  Max Henrion,et al.  Propagating uncertainty in bayesian networks by probabilistic logic sampling , 1986, UAI.

[30]  Gregory F. Cooper,et al.  An Empirical Evaluation of a Randomized Algorithm for Probabilistic Inference , 1989, UAI.

[31]  Richard M. Karp,et al.  Monte-Carlo Approximation Algorithms for Enumeration Problems , 1989, J. Algorithms.