Variational learning for rectified factor analysis

Linear factor models with non-negativity constraints have received a great deal of interest in a number of problem domains. In existing approaches, positivity has often been associated with sparsity. In this paper we argue that sparsity of the factors is not always a desirable option, but certainly a technical limitation of the currently existing solutions. We then reformulate the problem in order to relax the sparsity constraint while retaining positivity. This is achieved by employing a rectification nonlinearity rather than a positively supported prior directly on the latent space. A variational learning procedure is derived for the proposed model and this is contrasted to existing related approaches. Both i.i.d. and first-order AR variants of the proposed model are provided and they are experimentally demonstrated with artificial data. Application to the analysis of galaxy spectra show the benefits of the method in a real-world astrophysical problem, where the existing approach is not a viable alternative.

[1]  Antti Honkela,et al.  Variational learning and bits-back coding: an information-theoretic view to Bayesian learning , 2004, IEEE Transactions on Neural Networks.

[2]  H. Sebastian Seung,et al.  Algorithms for Non-negative Matrix Factorization , 2000, NIPS.

[3]  Michael I. Jordan,et al.  An Introduction to Variational Methods for Graphical Models , 1999, Machine-mediated learning.

[4]  Brendan J. Frey,et al.  Variational Learning in Nonlinear Gaussian Belief Networks , 1999, Neural Computation.

[5]  Yen-Wei Chen,et al.  Ensemble learning for independent component analysis , 2006, Pattern Recognit..

[6]  Louisa Anne Nolan.  The star formation history of elliptical galaxies , 2002 .

[7]  Bo Wang,et al.  Convergence and Asymptotic Normality of Variational Bayesian Approximations for Expon , 2004, UAI.

[8]  G. Kauffmann,et al.  The formation and evolution of galaxies within merging dark matter haloes , 1993 .

[9]  Astronomy,et al.  A data-driven Bayesian approach for finding young stellar populations in early-type galaxies from their ultraviolet-optical spectra , 2005, astro-ph/0511503.

[10]  Michael I. Jordan.  Learning in Graphical Models , 1999, NATO ASI Series.

[11]  R. Darlington,et al.  Factor Analysis , 2008 .

[12]  Puragra Guhathakurta,et al.  The DEEP2 Galaxy Redshift Survey: Spectral Classification of Galaxies at z ∼ 1 , 2003, astro-ph/0305587.

[13]  Markus Harva,et al.  Hierarchical Variance Models of Image Sequences , 2004 .

[14]  Michael I. Jordan,et al.  Learning from Incomplete Data , 1994 .

[15]  H. Sebastian Seung,et al.  Learning the parts of objects by non-negative matrix factorization , 1999, Nature.

[16]  Hagai Attias,et al.  A Variational Bayesian Framework for Graphical Models , 1999 .

[17]  A. Kabán,et al.  A variational Bayesian method for rectified factor analysis , 2005, Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005..

[18]  Ole Winther,et al.  Mean-Field Approaches to Independent Component Analysis , 2002, Neural Computation.

[19]  Matthew J. Beal,et al.  Variational Bayesian learning of directed graphical models with hidden variables , 2006 .

[20]  D Charles,et al.  Modelling multiple-cause structure using rectification constraints. , 1998, Network.

[21]  P. Paatero,et al.  Positive matrix factorization: A non-negative factor model with optimal utilization of error estimates of data values† , 1994 .

[22]  Michael I. Jordan,et al.  Beyond Independent Components: Trees and Clusters , 2003, J. Mach. Learn. Res..

[23]  Harri Lappalainen,et al.  Ensemble learning for independent component analysis , 1999 .

[24]  Seungjin Choi,et al.  Sequential EM learning for subspace analysis , 2002, Pattern Recognit. Lett..

[25]  Erkki Oja,et al.  A "nonnegative PCA" algorithm for independent component analysis , 2004, IEEE Transactions on Neural Networks.

[26]  Charles M. Bishop,et al.  Variational Message Passing , 2005, J. Mach. Learn. Res..

[27]  H. Sebastian Seung,et al.  The Rectified Gaussian Distribution , 1997, NIPS.

[28]  Puragra Guhathakurta,et al.  THE DEEP 2 GALAXY REDSHIFT SURVEY : SPECTRAL CLASSIFICATION OF GALAXIES AT Z ∼ 1 , 2008 .

[29]  Geoffrey E. Hinton,et al.  Generative models for discovering sparse distributed representations. , 1997, Philosophical transactions of the Royal Society of London. Series B, Biological sciences.