Parameter Expansion for Data Augmentation

Abstract Viewing the observed data of a statistical model as incomplete and augmenting its missing parts are useful for clarifying concepts and central to the invention of two well-known statistical algorithms: expectation-maximization (EM) and data augmentation. Recently, Liu, Rubin, and Wu demonstrated that expanding the parameter space along with augmenting the missing data is useful for accelerating iterative computation in an EM algorithm. The main purpose of this article is to rigorously define a parameter expanded data augmentation (PX-DA) algorithm and to study its theoretical properties. The PX-DA is a special way of using auxiliary variables to accelerate Gibbs sampling algorithms and is closely related to reparameterization techniques. We obtain theoretical results concerning the convergence rate of the PX-DA algorithm and the choice of prior for the expansion parameter. To understand the role of the expansion parameter, we establish a new theory for iterative conditional sampling under the tra...

[1]  K. Pearson.  Biometrika , 1902, The American Naturalist.

[2]  Richard S. Palais,et al.  On the Existence of Slices for Actions of Non-Compact Lie Groups , 1961 .

[3]  Donald Fraser,et al.  The fiducial method and invariance , 1961 .

[4]  Existence of local cross-sections in linear Cartan $G$-spaces under the action of noncompact groups , 1966 .

[5]  Structural Distributions Without Exact Transitivity , 1972 .

[6]  James V. Bondar.  Borel Cross-Sections and Maximal Invariants , 1976 .

[7]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[8]  Donald B. Rubin,et al.  Bayesian Inference for Causal Effects: The Role of Randomization , 1978 .

[9]  W. Wong,et al.  The calculation of posterior distributions by data augmentation , 1987 .

[10]  A. O'Hagan,et al.  The Calculation of Posterior Distributions by Data Augmentation: Comment , 1987 .

[11]  Roderick J. A. Little,et al.  Statistical Analysis with Missing Data , 1988 .

[12]  Adrian F. M. Smith,et al.  Sampling-Based Approaches to Calculating Marginal Densities , 1990 .

[13]  Charles J. Geyer,et al.  Practical Markov Chain Monte Carlo , 1992 .

[14]  Jun S. Liu,et al.  Covariance structure of the Gibbs sampler with applications to the comparisons of estimators and augmentation schemes , 1994 .

[15]  Jun S. Liu,et al.  The Collapsed Gibbs Sampler in Bayesian Computations with Applications to a Gene Regulation Problem , 1994 .

[16]  A. Gelfand,et al.  Efficient parametrisations for normal linear mixed models , 1995 .

[17]  D. Rubin,et al.  Parameter expansion to accelerate EM : The PX-EM algorithm , 1997 .

[18]  Xiao-Li Meng,et al.  The EM Algorithm—an Old Folk‐song Sung to a Fast New Tune , 1997 .

[19]  Jun S. Liu,et al.  Simulated Sintering : Markov Chain Monte Carlo With Spaces of Varying Dimensions , 1998 .

[20]  D. Higdon.  Auxiliary Variable Methods for Markov Chain Monte Carlo with Applications , 1998 .

[21]  D. Rubin,et al.  Parameter expansion to accelerate EM: The PX-EM algorithm , 1998 .

[22]  Xiao-Li Meng,et al.  Seeking efficient data augmentation schemes via conditional and marginal augmentation , 1999 .

[23]  A. U.S.,et al.  Generalised Gibbs sampler and multigrid Monte Carlo for Bayesian computation , 2000 .

[24]  Xiao-Li Meng,et al.  The Art of Data Augmentation , 2001 .