Tutorial on Online User Engagement: Metrics and Optimization

User engagement plays a central role in companies operating online services, such as search engines, news portals, e-commerce sites, entertainment services, and social networks. A main challenge is to leverage collected knowledge about the daily online behavior of millions of users to understand what engage them short-term and more importantly long-term. Two critical steps of improving user engagement are metrics and their optimization. The most common way that engagement is measured is through various online metrics, acting as proxy measures of user engagement. This tutorial will review these metrics, their advantages and drawbacks, and their appropriateness to various types of online services. Once metrics are defined, how to optimize them will become the key issue. We will survey methodologies including machine learning models and experimental designs that are utilized to optimize these metrics via direct or indirect ways. As case studies, we will focus on four types of services, news, search, entertainment, and e-commerce. We will end with lessons learned and a discussion on the most promising research directions.

[1]  Lihong Li,et al.  Toward Predicting the Outcome of an A/B Experiment for Search Relevance , 2015, WSDM.

[2]  Mounia Lalmas,et al.  Models of user engagement , 2012, UMAP.

[3]  Liangjie Hong,et al.  Returning is Believing: Optimizing Long-term User Engagement in Recommender Systems , 2017, CIKM.

[4]  Mike Thelwall,et al.  Synthesis Lectures on Information Concepts, Retrieval, and Services , 2009 .

[5]  Mounia Lalmas,et al.  Absence time and user engagement: evaluating ranking functions , 2013, WSDM '13.

[6]  Thorsten Joachims,et al.  Counterfactual Risk Minimization: Learning from Logged Bandit Feedback , 2015, ICML.

[7]  Suju Rajan,et al.  Learning Optimal Card Ranking from Query Reformulation , 2016, ArXiv.

[8]  Qian Zhao,et al.  GB-CENT: Gradient Boosted Categorical Embedding and Numerical Trees , 2017, WWW.

[9]  Fernando Diaz,et al.  Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems , 2018, CIKM.

[10]  Mounia Lalmas,et al.  Measuring User Engagement , 2014, Measuring User Engagement.

[11]  Yizhou Sun,et al.  On Sampling Strategies for Neural Network-based Collaborative Filtering , 2017, KDD.

[12]  Mounia Lalmas,et al.  Reader preferences and behavior on Wikipedia , 2014, HT.

[13]  Gabriella Kazai,et al.  Towards a science of user engagement. , 2011 .

[14]  Mounia Lalmas,et al.  Understanding User Attention and Engagement in Online News Reading , 2016, WSDM.

[15]  John Langford,et al.  Off-policy evaluation for slate recommendation , 2016, NIPS.

[16]  Huan Liu,et al.  Turning Clicks into Purchases: Revenue Optimization for Product Search in E-Commerce , 2018, SIGIR.

[17]  Suju Rajan,et al.  Beyond clicks: dwell time for personalization , 2014, RecSys '14.

[18]  Naren Ramakrishnan,et al.  A Gradient-based Adaptive Learning Framework for Efficient Personal Recommendation , 2017, RecSys.

[19]  Mounia Lalmas,et al.  Tutorial on Metrics of User Engagement: Applications to News, Search and E-Commerce , 2018, WSDM.

[20]  Fabrizio Silvestri,et al.  Improving Post-Click User Engagement on Native Ads via Survival Analysis , 2016, WWW.

[21]  Thorsten Joachims,et al.  Counterfactual Evaluation and Learning for Search, Recommendation and Ad Placement , 2016, SIGIR.

[22]  Thorsten Joachims,et al.  Unbiased Comparative Evaluation of Ranking Functions , 2016, ICTIR.

[23]  Ben Carterette,et al.  Mixed methods for evaluating user satisfaction , 2018, RecSys.

[24]  Marc Najork,et al.  Learning to Rank with Selection Bias in Personal Search , 2016, SIGIR.