Critically Examining the Claimed Value of Convolutions over User-Item Embedding Maps for Recommender Systems

In recent years, algorithm research in the area of recommender systems has shifted from matrix factorization techniques and their latent factor models to neural approaches. However, given the proven power of latent factor models, some newer neural approaches incorporate them within more complex network architectures. One specific idea, recently put forward by several researchers, is to consider potential correlations between the latent factors, i.e., embeddings, by applying convolutions over the user-item interaction map. However, contrary to what is claimed in these articles, such interaction maps do not share the properties of images where Convolutional Neural Networks (CNNs) are particularly useful. In this work, we show through analytical considerations and empirical evaluations that the claimed gains reported in the literature cannot be attributed to the ability of CNNs to model embedding correlations, as argued in the original papers. Moreover, additional performance evaluations show that all of the examined recent CNN-based models are outperformed by existing non-neural machine learning techniques or traditional nearest-neighbor approaches. On a more general level, our work points to major methodological issues in recommender systems research.

[1]  Lars Schmidt-Thieme,et al.  BPR: Bayesian Personalized Ranking from Implicit Feedback , 2009, UAI.

[2]  Yi Tay,et al.  Deep Learning based Recommender System: A Survey and New Perspectives , 2018 .

[3]  Dietmar Jannach,et al.  Evaluation of session-based recommendation algorithms , 2018, User Modeling and User-Adapted Interaction.

[4]  Matthew W. Hoffman,et al.  Predictive Entropy Search for Efficient Global Optimization of Black-box Functions , 2014, NIPS.

[5]  Xiangnan He,et al.  CFM: Convolutional Factorization Machines for Context-Aware Recommendation , 2019, IJCAI.

[6]  John Riedl,et al.  Item-based collaborative filtering recommendation algorithms , 2001, WWW '01.

[7]  Alistair Moffat,et al.  Improvements that don't add up: ad-hoc retrieval results since 1998 , 2009, CIKM.

[8]  Longbing Cao,et al.  CoupledCF: Learning Explicit and Implicit User-item Couplings in Recommendation for Deep Collaborative Filtering , 2018, IJCAI.

[9]  Walid Krichene,et al.  Neural Collaborative Filtering vs. Matrix Factorization Revisited , 2020, RecSys.

[10]  Evangelos Spiliotis,et al.  Statistical and Machine Learning forecasting methods: Concerns and ways forward , 2018, PloS one.

[11]  Ke Wang,et al.  Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding , 2018, WSDM.

[12]  John Riedl,et al.  GroupLens: an open architecture for collaborative filtering of netnews , 1994, CSCW '94.

[13]  Noemi Mauro,et al.  Performance comparison of neural and non-neural approaches to session-based recommendation , 2019, RecSys.

[14]  Xiaoyu Du,et al.  Outer Product-based Neural Collaborative Filtering , 2018, IJCAI.

[15]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[16]  Joseph F. Murray,et al.  Convolutional Networks Can Learn to Generate Affinity Graphs for Image Segmentation , 2010, Neural Computation.

[17]  Dietmar Jannach,et al.  A Troubling Analysis of Reproducibility and Progress in Recommender Systems Research , 2019, ACM Trans. Inf. Syst..

[18]  Roberto Turrin,et al.  Performance of recommender algorithms on top-n recommendation tasks , 2010, RecSys '10.

[19]  Yoshua Bengio,et al.  Gradient-based learning applied to document recognition , 1998, Proc. IEEE.

[20]  Matthew D. Hoffman,et al.  Variational Autoencoders for Collaborative Filtering , 2018, WWW.

[21]  George Karypis,et al.  SLIM: Sparse Linear Methods for Top-N Recommender Systems , 2011, 2011 IEEE 11th International Conference on Data Mining.

[22]  James Bennett,et al.  The Netflix Prize , 2007 .

[23]  Yifan Hu,et al.  Collaborative Filtering for Implicit Feedback Datasets , 2008, 2008 Eighth IEEE International Conference on Data Mining.

[24]  Trevor Darrell,et al.  Fully Convolutional Networks for Semantic Segmentation , 2017, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[25]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[26]  Jimmy J. Lin,et al.  Critically Examining the "Neural Hype": Weak Baselines and the Additivity of Effectiveness Gains from Neural Ranking Models , 2019, SIGIR.

[27]  Zachary C. Lipton,et al.  Troubling Trends in Machine Learning Scholarship , 2018, ACM Queue.

[28]  Xiaoyu Du,et al.  Modeling Embedding Dimension Correlations via Convolutional Neural Collaborative Filtering , 2019, ACM Trans. Inf. Syst..

[29]  Vladlen Koltun,et al.  Multi-Scale Context Aggregation by Dilated Convolutions , 2015, ICLR.

[30]  Yehuda Koren,et al.  Advances in Collaborative Filtering , 2011, Recommender Systems Handbook.

[31]  Yanghua Xiao,et al.  Convolutional Gaussian Embeddings for Personalized Recommendation with Uncertainty , 2019, IJCAI.

[32]  Xavier Bresson,et al.  Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering , 2016, NIPS.

[33]  Steffen Rendle,et al.  Factorization Machines , 2010, 2010 IEEE International Conference on Data Mining.

[34]  Maurizio Ferrari Dacrema,et al.  Artist-driven layering and user's behaviour impact on recommendations in a playlist continuation scenario , 2018, RecSys Challenge.

[35]  Simon Haykin,et al.  GradientBased Learning Applied to Document Recognition , 2001 .

[36]  Honglak Lee,et al.  Convolutional deep belief networks for scalable unsupervised learning of hierarchical representations , 2009, ICML '09.

[37]  Paul Covington,et al.  Deep Neural Networks for YouTube Recommendations , 2016, RecSys.

[38]  Abraham Bernstein,et al.  Updatable, Accurate, Diverse, and Scalable Recommendations for Interactive Applications , 2016, ACM Trans. Interact. Intell. Syst..

[39]  Geoffrey E. Hinton,et al.  Dynamic Routing Between Capsules , 2017, NIPS.

[40]  Joemon M. Jose,et al.  A Simple Convolutional Generative Network for Next Item Recommendation , 2018, WSDM.

[41]  Haobo Zhang,et al.  Integrating an Attention Mechanism and Convolution Collaborative Filtering for Document Context-Aware Rating Prediction , 2019, IEEE Access.

[42]  Dietmar Jannach,et al.  Are we really making much progress? A worrying analysis of recent neural recommendation approaches , 2019, RecSys.

[43]  Cédric Archambeau,et al.  One-Pass Ranking Models for Low-Latency Product Recommendations , 2015, KDD.

[44]  Vladlen Koltun,et al.  An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling , 2018, ArXiv.

[45]  Benjamin Schrauwen,et al.  Deep content-based music recommendation , 2013, NIPS.

[46]  Yehuda Koren,et al.  On the Difficulty of Evaluating Baselines: A Study on Recommender Systems , 2019, ArXiv.

[47]  Colin Cooper,et al.  Random walks in recommender systems: exact computation and simulations , 2014, WWW.

[48]  Christopher C. Johnson Logistic Matrix Factorization for Implicit Feedback Data , 2014 .