Understanding and Measuring User Engagement and Attention in Online News Reading

Prior work on user engagement with online news sites identified dwell time as a key engagement metric. Whereas on average, dwell time gives a reasonable estimate of user engagement with a news article, it does not capture user engagement with the news article at sub-document level nor it allows to measure the proportion of article read by the user. In this paper, we analyze online news reading patterns using large-scale viewport data collected from 267,210 page views on 1,971 news articles on a major online news website. We propose four engagement metrics that, unlike dwell time, more accurately reflect how users engage with and attend to the news content. The four metrics capture different levels of engagement, ranging from bounce to complete, providing clear and interpretable characterizations of user engagement with online news. Furthermore, we develop a probabilistic model that combines both an article textual content and level of user engagement information in a joint model. In our experiments we show that our model, called TUNE, is able to predict future level of user engagement based on textual content alone and outperform currently available methods.

[1]  Eugene Agichtein,et al.  Improving search result summaries by using searcher behavior data , 2013, SIGIR.

[2]  Alex Bateman,et al.  An introduction to hidden Markov models. , 2007, Current protocols in bioinformatics.

[3]  Elaine Toms,et al.  The development and evaluation of a survey to measure user engagement , 2010, J. Assoc. Inf. Sci. Technol..

[4]  J. Lafferty,et al.  Mixed-membership models of scientific publications , 2004, Proceedings of the National Academy of Sciences of the United States of America.

[5]  Mark Claypool,et al.  Implicit interest indicators , 2001, IUI '01.

[6]  Joemon M. Jose,et al.  User engagement in online News: Under the scope of sentiment, interest, affect, and gaze , 2014, J. Assoc. Inf. Sci. Technol..

[7]  Jude W. Shavlik,et al.  Learning users' interests by unobtrusively observing their normal behavior , 2000, IUI '00.

[8]  Eugene Agichtein,et al.  ViewSer: enabling large-scale remote user studies of web search examination and interaction , 2011, SIGIR.

[9]  Jeff Huang.  Web User Interaction Mining from Touch-Enabled Mobile Devices , 2012 .

[10]  Andreas Dengel,et al.  Segment-level display time as implicit feedback: a comparison to eye tracking , 2009, SIGIR.

[11]  Chih-Hung Hsieh,et al.  Towards better measurement of attention and satisfaction in mobile search , 2014, SIGIR.

[12]  Ling Xia,et al.  Eye tracking and online search: Lessons learned and challenges ahead , 2008, J. Assoc. Inf. Sci. Technol..

[13]  Mounia Lalmas,et al.  On saliency, affect and focused attention , 2012, CHI.

[14]  Suju Rajan,et al.  Beyond clicks: dwell time for personalization , 2014, RecSys '14.

[15]  Yunfei Chen,et al.  Evaluating the visual quality of web pages using a computational aesthetic approach , 2011, WSDM '11.

[16]  Gabriella Kazai,et al.  Towards a science of user engagement (Position Paper) , 2011 .

[17]  Alexander J. Smola,et al.  Measurement and modeling of eye-mouse behavior in the presence of nonlinear page layouts , 2013, WWW.

[18]  References , 1971 .

[19]  Eugene Agichtein,et al.  Beyond dwell time: estimating document relevance from cursor movements and other post-click searcher behavior , 2012, WWW.

[20]  Bracha Shapira,et al.  Study of the usefulness of known and new implicit indicators and their optimal combination for accurate inference of users interests , 2006, SAC.

[21]  Nick Craswell,et al.  An experimental comparison of click position-bias models , 2008, WSDM '08.

[22]  George Valkanas,et al.  Understanding Within-Content Engagement through Pattern Analysis of Mouse Gestures , 2014, CIKM.

[23]  Steve Fox,et al.  Evaluating implicit measures to improve web search , 2005, TOIS.

[24]  D. Rubin,et al.  Maximum likelihood from incomplete data via the EM - algorithm plus discussions on the paper , 1977 .

[25]  Ryen W. White,et al.  Modeling dwell time to predict click-level satisfaction , 2014, WSDM.

[26]  Eugene Agichtein,et al.  Discovering common motifs in cursor movement data for improving web search , 2014, WSDM.

[27]  Andrew McCallum,et al.  Topics over time: a non-Markov continuous-time model of topical trends , 2006, KDD '06.

[28]  Andrew McCallum,et al.  Topic Models Conditioned on Arbitrary Features with Dirichlet-multinomial Regression , 2008, UAI.

[29]  John R. Anderson,et al.  What can a mouse cursor tell us more?: correlation of eye/mouse movements on web browsing , 2001, CHI Extended Abstracts.

[30]  Susan T. Dumais,et al.  Improving Web Search Ranking by Incorporating User Behavior Information , 2019, SIGIR Forum.

[31]  Yoichi Shinoda,et al.  Information filtering based on user behavior analysis and best match text retrieval , 1994, SIGIR '94.

[32]  T. Minka.  Estimating a Dirichlet distribution , 2012 .

[33]  Ryen W. White,et al.  Mining the search trails of surfing crowds: identifying relevant websites from user activity , 2008, WWW.

[34]  Michael I. Jordan,et al.  Latent Dirichlet Allocation , 2001, J. Mach. Learn. Res..

[35]  Ryen W. White,et al.  Understanding web browsing behaviors through Weibull analysis of dwell time , 2010, SIGIR.