Memory-efficient Embedding for Recommendations

Practical large-scale recommender systems usually contain thousands of feature fields from users, items, contextual information, and their interactions. Most of them empirically allocate a unified dimension to all feature fields, which is memory inefficient. Thus it is highly desired to assign different embedding dimensions to different feature fields according to their importance and predictability. Due to the large amounts of feature fields and the nuanced relationship between embedding dimensions with feature distributions and neural network architectures, manually allocating embedding dimensions in practical recommender systems can be very difficult. To this end, we propose an AutoML based framework (AutoDim) in this paper, which can automatically select dimensions for different feature fields in a data-driven fashion. Specifically, we first proposed an end-to-end differentiable framework that can calculate the weights over various dimensions for feature fields in a soft and continuous manner with an AutoML based optimization algorithm; then we derive a hard and discrete embedding component architecture according to the maximal weights and retrain the whole recommender framework. We conduct extensive experiments on benchmark datasets to validate the effectiveness of the AutoDim framework.

[1]  Steffen Rendle,et al.  Factorization Machines , 2010, 2010 IEEE International Conference on Data Mining.

[2]  Ben Poole,et al.  Categorical Reparameterization with Gumbel-Softmax , 2016, ICLR.

[3]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.

[4]  Huan Liu,et al.  What Your Images Reveal: Exploiting Visual Contents for Point-of-Interest Recommendation , 2017, WWW.

[5]  Jiliang Tang,et al.  AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations , 2020, ArXiv.

[6]  Sergey Ioffe,et al.  Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift , 2015, ICML.

[7]  Jian Tang,et al.  AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks , 2018, CIKM.

[8]  Lior Rokach,et al.  Introduction to Recommender Systems Handbook , 2011, Recommender Systems Handbook.

[9]  Li Fei-Fei,et al.  Progressive Neural Architecture Search , 2017, ECCV.

[10]  Song Han,et al.  Path-Level Network Transformation for Efficient Architecture Search , 2018, ICML.

[11]  Yu Sun,et al.  Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising , 2018, WWW.

[12]  Yiming Yang,et al.  DARTS: Differentiable Architecture Search , 2018, ICLR.

[13]  Tat-Seng Chua,et al.  Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks , 2017, IJCAI.

[14]  Quoc V. Le,et al.  Neural Input Search for Large Scale Recommendation Models , 2019, KDD.

[15]  David Heckerman,et al.  Empirical Analysis of Predictive Algorithms for Collaborative Filtering , 1998, UAI.

[16]  Quoc V. Le,et al.  Efficient Neural Architecture Search via Parameter Sharing , 2018, ICML.

[17]  Yunming Ye,et al.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction , 2017, IJCAI.

[18]  Yi Tay,et al.  Deep Learning based Recommender System: A Survey and New Perspectives , 2018 .

[19]  Tat-Seng Chua,et al.  Neural Factorization Machines for Sparse Predictive Analytics , 2017, SIGIR.

[20]  Vijay Vasudevan,et al.  Learning Transferable Architectures for Scalable Image Recognition , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[21]  James Zou,et al.  Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems , 2019, 2021 IEEE International Symposium on Information Theory (ISIT).

[22]  Jun Wang,et al.  Product-Based Neural Networks for User Response Prediction , 2016, 2016 IEEE 16th International Conference on Data Mining (ICDM).

[23]  Alok Aggarwal,et al.  Regularized Evolution for Image Classifier Architecture Search , 2018, AAAI.

[24]  Wei Wu,et al.  Practical Block-Wise Neural Network Architecture Generation , 2017, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[25]  Mohamed F. Mokbel,et al.  Recommendations in location-based social networks: a survey , 2015, GeoInformatica.

[26]  Yong Liu,et al.  Improved Recurrent Neural Networks for Session-based Recommendations , 2016, DLRS@RecSys.

[27]  Khaled Shaalan,et al.  A Survey of Web Information Extraction Systems , 2006, IEEE Transactions on Knowledge and Data Engineering.

[28]  Theodore Lim,et al.  SMASH: One-Shot Model Architecture Search through HyperNetworks , 2017, ICLR.

[29]  Lars Schmidt-Thieme,et al.  Personalized Deep Learning for Tag Recommendation , 2017, PAKDD.

[30]  Bin Liu,et al.  AutoFIS: Automatic Feature Interaction Selection in Factorization Models for Click-Through Rate Prediction , 2020, KDD.

[31]  Bo Chen,et al.  MnasNet: Platform-Aware Neural Architecture Search for Mobile , 2018, 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[32]  Gang Chen,et al.  Personal recommendation using deep recurrent neural networks in NetEase , 2016, 2016 IEEE 32nd International Conference on Data Engineering (ICDE).

[33]  Greg Linden,et al.  Amazon . com Recommendations Item-to-Item Collaborative Filtering , 2001 .

[34]  Guorui Zhou,et al.  Deep Interest Network for Click-Through Rate Prediction , 2017, KDD.

[35]  Xiangnan He,et al.  Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention , 2017, SIGIR.

[36]  Yizhou Sun,et al.  Differentiable Product Quantization for End-to-End Embedding Compression , 2019, ICML.

[37]  E. Gumbel.  Statistical Theory of Extreme Values and Some Practical Applications : A Series of Lectures , 1954 .

[38]  Shuai Yang,et al.  FuxiCTR: An Open Benchmark for Click-Through Rate Prediction , 2020, ArXiv.

[39]  Jun Wang,et al.  Deep Learning over Multi-field Categorical Data - - A Case Study on User Response Prediction , 2016, ECIR.

[40]  Paul Covington,et al.  Deep Neural Networks for YouTube Recommendations , 2016, RecSys.

[41]  Gang Fu,et al.  Deep & Cross Network for Ad Click Predictions , 2017, ADKDD@KDD.

[42]  Peng Zhang,et al.  IRGAN: A Minimax Game for Unifying Generative and Discriminative Information Retrieval Models , 2017, SIGIR.

[43]  Sheng Li,et al.  Deep Collaborative Filtering via Marginalized Denoising Auto-encoder , 2015, CIKM.

[44]  Tie-Yan Liu,et al.  Neural Architecture Optimization , 2018, NeurIPS.

[45]  Loriene Roy,et al.  Content-based book recommending using learning for text categorization , 1999, DL '00.

[46]  Alexandros Karatzoglou,et al.  Session-based Recommendations with Recurrent Neural Networks , 2015, ICLR.

[47]  Yanyan Shen,et al.  Differentiable Neural Input Search for Recommender Systems , 2020, ArXiv.

[48]  Xing Xie,et al.  xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems , 2018, KDD.

[49]  Sophie Ahrens,et al.  Recommender Systems , 2012 .

[50]  Quoc V. Le,et al.  Neural Architecture Search with Reinforcement Learning , 2016, ICLR.

[51]  Scott Sanner,et al.  AutoRec: Autoencoders Meet Collaborative Filtering , 2015, WWW.

[52]  Liang Lin,et al.  SNAS: Stochastic Neural Architecture Search , 2018, ICLR.

[53]  Dong Lin,et al.  Learning Multi-granular Quantized Embeddings for Large-Vocab Categorical Features in Recommender Systems , 2020, WWW.