Combating the Filter Bubble: Designing for Serendipity in a University Course Recommendation System

Collaborative filtering based algorithms, including Recurrent Neural Networks (RNN), tend towards predicting a perpetuation of past observed behavior. In a recommendation context, this can lead to an overly narrow set of suggestions lacking in serendipity and inadvertently placing the user in what is known as a "filter bubble." In this paper, we grapple with the issue of the filter bubble in the context of a course recommendation system in production at a public university. Our approach is to present course results that are novel or unexpected to the student but still relevant to their interests. We build one set of models based on course catalog descriptions (BOW) and another set informed by enrollment histories (course2vec). We compare the performance of these models on off-line validation sets and against the system's existing RNN-based recommendation engine in an online user study of undergraduates (N = 70) who rated their course recommendations along six characteristics related to serendipity. Results of the user study show a dramatic lack of novelty in RNN recommendations and depict the characteristic trade-offs that make serendipity difficult to achieve. While the machine learned course2vec models performed best on off-line validation tasks, it was the simple bag-of-words based recommendations that students rated as more serendipitous. We discuss the role of the kind of information presented by the system in a student's decision to accept a recommendation from either algorithm.

[1]  Daniel R. Figueiredo,et al.  struc2vec: Learning Node Representations from Structural Identity , 2017, KDD.

[2]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[3]  Pietro Perona,et al.  One-shot learning of object categories , 2006, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[4]  Alexandros Karatzoglou,et al.  Parallel Recurrent Neural Network Architectures for Feature-rich Session-based Recommendations , 2016, RecSys.

[5]  Zachary A. Pardos,et al.  Goal-based Course Recommendation , 2018, LAK.

[6]  Daniele Quercia,et al.  Auralist: introducing serendipity into music recommendation , 2012, WSDM '12.

[7]  Aditya G. Parameswaran,et al.  Recommendation systems with complex constraints: A course recommendation perspective , 2011, TOIS.

[8]  Omer Levy,et al.  Linguistic Regularities in Sparse and Explicit Word Representations , 2014, CoNLL.

[9]  Haiyi Zhu,et al.  Towards Value-Sensitive Learning Analytics Design , 2018, LAK.

[10]  Hung-Hsuan Chen,et al.  Behavior2Vec: Generating Distributed Representations of Users' Behaviors on Products for Recommender Systems , 2018, ACM Trans. Knowl. Discov. Data.

[11]  Andrew Abbott.  Department and Discipline: Chicago Sociology at One Hundred , 1999 .

[12]  Pasquale Lops,et al.  Introducing Serendipity in a Content-Based Recommender System , 2008, 2008 Eighth International Conference on Hybrid Intelligent Systems.

[13]  Ramesh Johari,et al.  How a data-driven course planning tool affects college students' GPA: evidence from two field experiments , 2018, L@S.

[14]  Takeshi Yamada,et al.  Personalized recommendation based on the personal innovator degree , 2009, RecSys '09.

[15]  Zachary A. Pardos,et al.  Design and Deployment of a Better Course Search Tool: Inferring Latent Keywords from Enrollment Networks , 2019, EC-TEL.

[16]  Sean M. McNee,et al.  Being accurate is not enough: how accuracy metrics have hurt recommender systems , 2006, CHI Extended Abstracts.

[17]  Zachary A. Pardos,et al.  Time slice imputation for personalized goal-based recommendation in higher education , 2019, RecSys.

[18]  Huzefa Rangwala,et al.  Grade Prediction Based on Cumulative Knowledge and Co-taken Courses , 2019, EDM.

[19]  Guy Shani,et al.  Evaluating Recommendation Systems , 2011, Recommender Systems Handbook.

[20]  PantelPatrick,et al.  From frequency to meaning , 2010 .

[21]  Mouzhi Ge,et al.  Beyond accuracy: evaluating recommender systems by coverage and serendipity , 2010, RecSys '10.

[22]  King-Sun Fu,et al.  IEEE Transactions on Pattern Analysis and Machine Intelligence Publication Information , 2004, IEEE Transactions on Pattern Analysis and Machine Intelligence.

[23]  Sergei Vassilvitskii,et al.  Getting recommender systems to think outside the box , 2009, RecSys '09.

[24]  Alessandro Lenci,et al.  Distributional Memory: A General Framework for Corpus-Based Semantics , 2010, CL.

[25]  Oriol Vinyals,et al.  Matching Networks for One Shot Learning , 2016, NIPS.

[26]  Ido Dagan,et al.  Directional distributional similarity for lexical inference , 2010, Natural Language Engineering.

[27]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[28]  Naftali Tishby,et al.  Distributional Clustering of English Words , 1993, ACL.

[29]  Georgiana Dinu,et al.  Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors , 2014, ACL.

[30]  Gaurav Pandey,et al.  Recommending Serendipitous Items using Transfer Learning , 2018, CIKM.

[31]  Zachary A. Pardos,et al.  A Map of Knowledge , 2018, ArXiv.

[32]  Isabelle Tellier,et al.  Exploring Vector Spaces for Semantic Relations , 2017, EMNLP.

[33]  George Karypis,et al.  Scholars Walk: A Markov Chain Framework for Course Recommendation , 2019, EDM.

[34]  M. Stevens,et al.  Association, Service, Market: Higher Education in American Political Development , 2016 .

[35]  Hinrich Schütze,et al.  Introduction to information retrieval , 2008 .

[36]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[37]  Dekang Lin,et al.  An Information-Theoretic Definition of Similarity , 1998, ICML.

[38]  Loren G. Terveen,et al.  Exploring the filter bubble: the effect of using recommender systems on content diversity , 2014, WWW.

[39]  Panagiotis Adamopoulos,et al.  On Unexpectedness in Recommender Systems , 2013, ACM Trans. Intell. Syst. Technol..

[40]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[41]  Zachary A. Pardos,et al.  Connectionist recommendation in the wild: on the utility and scrutability of neural networks for personalized course guidance , 2018, User Modeling and User-Adapted Interaction.

[42]  Peter Brusilovsky,et al.  Encouraging user participation in a course recommender system: An impact on user behavior , 2011, Comput. Hum. Behav..

[43]  Patrick Pantel,et al.  From Frequency to Meaning: Vector Space Models of Semantics , 2010, J. Artif. Intell. Res..

[44]  Martin Dillon,et al.  Introduction to modern information retrieval: G. Salton and M. McGill. McGraw-Hill, New York (1983). xv + 448 pp., $32.95 ISBN 0-07-054484-0 , 1983 .

[45]  Judy Kay,et al.  Intelligent Tutoring Systems , 2000, Lecture Notes in Computer Science.

[46]  Jürgen Ziegler,et al.  Impact of item consumption on assessment of recommendations in user studies , 2018, RecSys.