Beta Processes, Stick-Breaking and Power Laws

The beta-Bernoulli process provides a Bayesian nonparametric prior for models involving collections of binary-valued features. A draw from the beta process yields an infinite collection of probabilities in the unit interval, and a draw from the Bernoulli process turns these into binaryvalued features. Recent work has provided stick-breaking representations for the beta process analogous to the well-known stick-breaking representation for the Dirichlet process. We derive one such stick-breaking representation directly from the characterization of the beta process as a completely random measure. This approach motivates a three-parameter generalization of the beta process, and we study the power laws that can be obtained from this generalized beta process. We present a posterior inference algorithm for the beta-Bernoulli process that exploits the stickbreaking representation, and we present experimental results for a discrete factor-analysis model.

[1]  George Kingsley Zipf,et al.  Human Behaviour and the Principle of Least Effort: an Introduction to Human Ecology , 2012 .

[2]  A. Erdélyi,et al.  The asymptotic expansion of a ratio of gamma functions. , 1951 .

[3]  H. Chernoff.  A Measure of Asymptotic Efficiency for Tests of a Hypothesis Based on the sum of Observations , 1952 .

[4]  J. McCloskey,et al.  A model for the distribution of individuals by species in an environment , 1965 .

[5]  W. Feller,et al.  An Introduction to Probability Theory and its Applications, Vol. II , 1967 .

[6]  J. Kingman,et al.  Completely random measures. , 1967 .

[7]  W. Feller,et al.  An Introduction to Probability Theory and Its Applications , 1951 .

[8]  T. Ferguson.  A Bayesian Analysis of Some Nonparametric Problems , 1973 .

[9]  D. Freedman.  Another Note on the Borel-Cantelli Lemma and the Strong Law, with the Poisson Approximation as a By-product , 1973 .

[10]  H. S. Heaps,et al.  Information retrieval, computational and theoretical aspects , 1978 .

[11]  N. Hjort.  Nonparametric Bayes Estimators Based on Beta Processes in Models for Life History Data , 1990 .

[12]  Torben Hagerup,et al.  A Guided Tour of Chernoff Bounds , 1990, Inf. Process. Lett..

[13]  J. Sethuraman.  A CONSTRUCTIVE DEFINITION OF DIRICHLET PRIORS , 1991 .

[14]  J. Pitman,et al.  The two-parameter Poisson-Dirichlet distribution derived from a stable subordinator , 1997 .

[15]  S. MacEachern.  Decision Theoretic Aspects of Dependent Nonparametric Processes , 2000 .

[16]  Yongdai Kim,et al.  On posterior consistency of survival models , 2001 .

[17]  Lancelot F. James,et al.  Gibbs Sampling Methods for Stick-Breaking Priors , 2001 .

[18]  Michael Mitzenmacher,et al.  A Brief History of Generative Models for Power Law and Lognormal Distributions , 2004, Internet Math..

[19]  Reflecting uncertainty in inverse problems: a Bayesian solution using Lévy processes , 2004 .

[20]  Thomas L. Griffiths,et al.  Infinite latent feature models and the Indian buffet process , 2005, NIPS.

[21]  B. Schölkopf,et al.  Edinburgh Research Explorer Interpolating between types and tokens by estimating power-law generators , 2006 .

[22]  J. Pitman.  Combinatorial Stochastic Processes , 2006 .

[23]  Michael I. Jordan,et al.  Variational inference for Dirichlet process mixtures , 2006 .

[24]  Stephen G. Walker,et al.  Sampling the Dirichlet Mixture Model with Slices , 2006, Commun. Stat. Simul. Comput..

[25]  J. Pitman,et al.  Notes on the occupancy problem with infinitely many boxes: general asymptotics and power laws ∗ , 2007, math/0701718.

[26]  Massimo Franceschetti,et al.  Closing the Gap in the Capacity of Wireless Networks Via Percolation Theory , 2007, IEEE Transactions on Information Theory.

[27]  Michael I. Jordan,et al.  Hierarchical Beta Processes and the Indian Buffet Process , 2007, AISTATS.

[28]  Yee Whye Teh,et al.  Stick-breaking Construction for the Indian Buffet Process , 2007, AISTATS.

[29]  Lawrence Carin,et al.  A Stick-Breaking Construction of the Beta Process , 2010, ICML.

[30]  Stephen G. Walker,et al.  Slice sampling mixture models , 2011, Stat. Comput..

[31]  The Stick-Breaking Construction of the Beta Process as a Poisson Process , 2011, 1109.0343.