RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems

Robust machine learning is an increasingly important topic that focuses on developing models resilient to various forms of imperfect data. Due to the pervasiveness of recommender systems in online technologies, researchers have carried out several robustness studies focusing on data sparsity and profile injection attacks. Instead, we propose a more holistic view of robustness for recommender systems that encompasses multiple dimensions robustness with respect to sub-populations, transformations, distributional disparity, attack, and data sparsity. While there are several libraries that allow users to compare different recommender system models, there is no software library for comprehensive robustness evaluation of recommender system models under different scenarios. As our main contribution, we present a robustness evaluation toolkit, Robustness Gym for RecSys (RGRecSys),1 that allows us to quickly and uniformly evaluate the robustness of recommender system models.

[1]  Collin Jackson,et al.  Robust defenses for cross-site request forgery , 2008, CCS.

[2]  Li-zhen Cui,et al.  GCN-Based User Representation Learning for Unifying Robust Recommendation and Fraudster Detection , 2020, SIGIR.

[3]  Mingxuan Sun,et al.  PREA: personalized recommendation algorithms toolkit , 2012, J. Mach. Learn. Res..

[4]  Yunqi Li,et al.  User-oriented Fairness in Recommendation , 2021, WWW.

[5]  Xu Chen,et al.  Explainable Recommendation: A Survey and New Perspectives , 2018, Found. Trends Inf. Retr..

[6]  Ji-Rong Wen,et al.  RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms , 2020, CIKM.

[7]  Thorsten Joachims,et al.  Unbiased Learning-to-Rank with Biased Feedback , 2016, WSDM.

[8]  Hanxiong Chen,et al.  Neural Collaborative Reasoning , 2020, WWW.

[9]  Gang Fu,et al.  Deep & Cross Network for Ad Click Predictions , 2017, ADKDD@KDD.

[10]  Longbing Cao,et al.  Non-IID Recommender Systems: A Review and Framework of Recommendation Paradigm Shifting , 2016, ArXiv.

[11]  Elena Zheleva,et al.  Propensity-Independent Bias Recovery in Offline Learning-to-Rank Systems , 2021, SIGIR.

[12]  Fabio Aiolli,et al.  Efficient top-n recommendation for very large scale binary rated datasets , 2013, RecSys.

[13]  Neil Yorke-Smith,et al.  LibRec: A Java Library for Recommender Systems , 2015, UMAP Workshops.

[14]  Lei Zheng,et al.  Spectral collaborative filtering , 2018, RecSys.

[15]  Steffen Rendle,et al.  Factorization Machines , 2010, 2010 IEEE International Conference on Data Mining.

[16]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[17]  Thorsten Joachims,et al.  Fairness of Exposure in Rankings , 2018, KDD.

[18]  Weinan Zhang,et al.  Interactive Recommender System via Knowledge Graph-enhanced Reinforcement Learning , 2020, SIGIR.

[19]  F. Maxwell Harper,et al.  The MovieLens Datasets: History and Context , 2016, TIIS.

[20]  Yongdong Zhang,et al.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation , 2020, SIGIR.

[21]  Tat-Seng Chua,et al.  Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks , 2017, IJCAI.

[22]  Mohit Bansal,et al.  Robustness Gym: Unifying the NLP Evaluation Landscape , 2021, NAACL.

[23]  Zi Huang,et al.  From Zero-Shot Learning to Cold-Start Recommendation , 2019, AAAI.

[24]  Matthew Richardson,et al.  Predicting clicks: estimating the click-through rate for new ads , 2007, WWW '07.

[25]  Yifan Zhang,et al.  Correcting for Selection Bias in Learning-to-rank Systems , 2020, WWW.

[26]  Larry P. Heck,et al.  Learning deep structured semantic models for web search using clickthrough data , 2013, CIKM.

[27]  John Riedl,et al.  Rethinking the recommender research ecosystem: reproducibility, openness, and LensKit , 2011, RecSys '11.

[28]  Peng Cui,et al.  Towards Out-Of-Distribution Generalization: A Survey , 2021, ArXiv.