Singular value decomposition (SVD) is a matrix decomposition algorithm that returns the optimal (in the sense of squared error) low-rank decomposition of a matrix. SVD has found widespread use across a variety of machine learning applications, where its output is interpreted as compact and informative representations of data. The Netflix Prize challenge, and collaborative filtering in general, is an ideal application for SVD, since the data is a matrix of ratings given by users to movies. It is thus not surprising to observe that most currently successful teams use SVD, either with an extension, or to interpolate with results returned by other algorithms. Unfortunately SVD can easily overfit due to the extreme data sparsity of the matrix in the Netflix Prize challenge, and care must be taken to regularize properly. In this paper, we propose a Bayesian approach to alleviate overfitting in SVD, where priors are introduced and all parameters are integrated out using variational inference. We show experimentally that this gives significantly improved results over vanilla SVD. For truncated SVDs of rank 5, 10, 20, and 30, our proposed Bayesian approach achieves 2.2% improvement over a näıve approach, 1.6% improvement over a gradient descent approach dealing with unobserved entries properly, and 0.9% improvement over a maximum a posteriori (MAP) approach.
[1]
Radford M. Neal.
Assessing Relevance determination methods using DELVE
,
1998
.
[2]
William H. Press,et al.
Numerical recipes in C
,
2002
.
[3]
Matthew J. Beal.
Variational algorithms for approximate Bayesian inference
,
2003
.
[4]
Tommi S. Jaakkola,et al.
Weighted Low-Rank Approximations
,
2003,
ICML.
[5]
Matthew Brand,et al.
Fast Online SVD Revisions for Lightweight Recommender Systems
,
2003,
SDM.
[6]
Benjamin M. Marlin,et al.
Collaborative Filtering: A Machine Learning Perspective
,
2004
.
[7]
Nathan Srebro,et al.
Fast maximum margin matrix factorization for collaborative prediction
,
2005,
ICML.
[8]
Zoubin Ghahramani,et al.
Modeling Dyadic Data with Binary Latent Factors
,
2006,
NIPS.
[9]
Dennis DeCoste,et al.
Collaborative prediction using ensembles of Maximum Margin Matrix Factorizations
,
2006,
ICML.
[10]
James Bennett,et al.
The Netflix Prize
,
2007
.