Connectionist recommendation in the wild: on the utility and scrutability of neural networks for personalized course guidance

The aggregate behaviors of users can collectively encode deep semantic information about the objects with which they interact. In this paper, we demonstrate novel ways in which the synthesis of these data can illuminate the terrain of users’ environment and support them in their decision making and wayfinding. A novel application of recurrent neural networks and skip-gram models, approaches popularized by their application to modeling language, are brought to bear on student university enrollment sequences to create vector representations of courses and map out traversals across them. We present demonstrations of how scrutability from these neural networks can be gained and how the combination of these techniques can be seen as an evolution of content tagging and a means for a recommender to balance user preferences inferred from data with those explicitly specified. From validation of the models to the development of a UI, we discuss additional requisite functionality informed by the results of a usability study leading to the ultimate deployment of the system at a university.

[1]  Zachary A. Pardos,et al.  A Map of Knowledge , 2018, ArXiv.

[2]  Geoffrey E. Hinton,et al.  Learning distributed representations of concepts. , 1989 .

[3]  Greg Linden,et al.  Amazon . com Recommendations Item-to-Item Collaborative Filtering , 2001 .

[4]  Zachary A. Pardos,et al.  Diagnosing University Student Subject Proficiency and Predicting Degree Completion in Vector Space , 2018, AAAI.

[5]  Jürgen Schmidhuber,et al.  Long Short-Term Memory , 1997, Neural Computation.

[6]  Matthew D. Pistilli,et al.  Course signals at Purdue: using learning analytics to increase student success , 2012, LAK.

[7]  Aditya G. Parameswaran,et al.  Recommendation systems with complex constraints: A course recommendation perspective , 2011, TOIS.

[8]  Qi Gao,et al.  Analyzing user modeling on twitter for personalized news recommendations , 2011, UMAP'11.

[9]  Laura K. Allen,et al.  Does agency matter?: Exploring the impact of controlled behaviors within a game-based environment , 2015, Comput. Educ..

[10]  John Riedl,et al.  Recommender systems: from algorithms to user experience , 2012, User Modeling and User-Adapted Interaction.

[11]  Hugues Bersini,et al.  Long and Short-Term Recommendations with Recurrent Neural Networks , 2017, UMAP.

[12]  Lise Getoor,et al.  User Preferences for Hybrid Explanations , 2017, RecSys.

[13]  Dragan Gasevic,et al.  Open Learning Analytics: an integrated modularized platform , 2011 .

[14]  Fabiana Vernero,et al.  SoNARS: A Social Networks-Based Algorithm for Social Recommender Systems , 2009, UMAP.

[15]  Barry Smyth,et al.  Using twitter to recommend real-time topical news , 2009, RecSys '09.

[16]  Patrick Gallinari,et al.  Latent Trajectory Modeling: A Light and Efficient Way to Introduce Time in Recommender Systems , 2015, RecSys.

[17]  Razvan Pascanu,et al.  Theano: new features and speed improvements , 2012, ArXiv.

[18]  Bamshad Mobasher,et al.  Recommender Systems as Multistakeholder Environments , 2017, UMAP.

[19]  Jonathan Gemmell,et al.  Infusing Collaborative Recommenders with Distributed Representations , 2016, DLRS@RecSys.

[20]  Geoffrey E. Hinton,et al.  ImageNet classification with deep convolutional neural networks , 2012, Commun. ACM.

[21]  Zachary A. Pardos,et al.  The School of Information and its relationship to computer science at UC Berkeley , 2017 .

[22]  Daniel Thalmann,et al.  A simple but effective method to incorporate trusted neighbors in recommender systems , 2012, UMAP.

[23]  Jacob Whitehill,et al.  Delving Deeper into MOOC Student Dropout Prediction , 2017, ArXiv.

[24]  Guillermo Sapiro,et al.  Online Learning for Matrix Factorization and Sparse Coding , 2009, J. Mach. Learn. Res..

[25]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[26]  Wei Xu,et al.  Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) , 2014, ICLR.

[27]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[28]  Omer Levy,et al.  Neural Word Embedding as Implicit Matrix Factorization , 2014, NIPS.

[29]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[30]  Pasquale Lops,et al.  Learning Word Embeddings from Wikipedia for Content-Based Recommender Systems , 2016, ECIR.

[31]  Hung-Hsuan Chen,et al.  Behavior2Vec: Generating Distributed Representations of Users' Behaviors on Products for Recommender Systems , 2018, ACM Trans. Knowl. Discov. Data.

[32]  Peter Brusilovsky,et al.  Methods and techniques of adaptive hypermedia , 1996, User Modeling and User-Adapted Interaction.

[33]  Omer Levy,et al.  Dependency-Based Word Embeddings , 2014, ACL.

[34]  Albert T. Corbett,et al.  Preface to the UMUAI special issue on the impact of learner modeling , 2017, User Modeling and User-Adapted Interaction.

[35]  Jun Zhao,et al.  Recurrent Convolutional Neural Networks for Text Classification , 2015, AAAI.

[36]  Zachary A. Pardos,et al.  Enabling Real-Time Adaptivity in MOOCs with a Personalized Next-Step Recommendation Framework , 2017, L@S.

[37]  Lukás Burget,et al.  Extensions of recurrent neural network language model , 2011, 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).

[38]  Jenna Burrell,et al.  How the machine ‘thinks’: Understanding opacity in machine learning algorithms , 2016 .

[39]  Yoshua Bengio,et al.  Show, Attend and Tell: Neural Image Caption Generation with Visual Attention , 2015, ICML.

[40]  A. Diamond,et al.  Interventions Shown to Aid Executive Function Development in Children 4 to 12 Years Old , 2011, Science.

[41]  Frank K. Soong,et al.  TTS synthesis with bidirectional LSTM based recurrent neural networks , 2014, INTERSPEECH.

[42]  Petr Sojka,et al.  Software Framework for Topic Modelling with Large Corpora , 2010 .

[43]  Robin van Meteren Using Content-Based Filtering for Recommendation , 2000 .

[44]  George Karypis,et al.  Domain-Aware Grade Prediction and Top-n Course Recommendation , 2016, RecSys.

[45]  Patrick Seemann,et al.  Matrix Factorization Techniques for Recommender Systems , 2014 .

[46]  Carolyn Penstein Rosé,et al.  “ Turn on , Tune in , Drop out ” : Anticipating student dropouts in Massive Open Online Courses , 2013 .

[47]  Robin D. Burke,et al.  Hybrid Recommender Systems: Survey and Experiments , 2002, User Modeling and User-Adapted Interaction.

[48]  Dan Frankowski,et al.  Collaborative Filtering Recommender Systems , 2007, The Adaptive Web.

[49]  Dietmar Jannach,et al.  User Modeling and User-Adapted Interaction Session-based Item Recommendation in E-Commerce On Short-Term Intents , Reminders , Trends , and Discounts , 2017 .

[50]  Zhen Li,et al.  Visual planner: beyond prerequisites, designing an interactive course planner for a 21st century flexible curriculum , 2012, CHI EA '12.

[51]  David M. Blei,et al.  Modeling User Exposure in Recommendation , 2015, WWW.

[52]  Razvan Pascanu,et al.  Theano: A CPU and GPU Math Compiler in Python , 2010, SciPy.

[53]  Xingshe Zhou,et al.  TV Program Recommendation for Multiple Viewers Based on user Profile Merging , 2006, User Modeling and User-Adapted Interaction.

[54]  Vincent Aleven,et al.  Instruction Based on Adaptive Learning Technologies , 2016 .

[55]  Zachary A. Pardos,et al.  Analysis of Student Behaviour in Habitable Worlds Using Continuous Representation Visualization , 2017, ArXiv.

[56]  Peter Brusilovsky,et al.  Encouraging user participation in a course recommender system: An impact on user behavior , 2011, Comput. Hum. Behav..

[57]  Judy Kay,et al.  Scrutability as a core interface element , 2005, AIED.

[58]  Tat-Seng Chua,et al.  TEM: Tree-enhanced Embedding Model for Explainable Recommendation , 2018, WWW.

[59]  Jacob Whitehill,et al.  MOOC Dropout Prediction: How to Measure Accuracy? , 2017, L@S.

[60]  Jürgen Schmidhuber,et al.  Learning to forget: continual prediction with LSTM , 1999 .

[61]  Yoshua Bengio,et al.  A Neural Probabilistic Language Model , 2003, J. Mach. Learn. Res..

[62]  Alexandros Karatzoglou,et al.  Session-based Recommendations with Recurrent Neural Networks , 2015, ICLR.

[63]  Zachary A. Pardos,et al.  Predictive Modelling of Student Behavior Using Granular Large-Scale Action Data , 2017 .

[64]  Zachary A. Pardos,et al.  Distributed Representation of Misconceptions , 2018, ICLS.

[65]  Michael J. Pazzani,et al.  A Framework for Collaborative, Content-Based and Demographic Filtering , 1999, Artificial Intelligence Review.

[66]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[67]  Afet Dundar,et al.  Completing College: A National View of Student Completion Rates--Fall 2011 Cohort. (Signature Report No. 14). , 2017 .

[68]  Alex Graves,et al.  DRAW: A Recurrent Neural Network For Image Generation , 2015, ICML.

[69]  Ramesh Johari,et al.  How a data-driven course planning tool affects college students' GPA: evidence from two field experiments , 2018, L@S.

[70]  Eitel J. M. Lauría,et al.  Early Alert of Academically At-Risk Students: An Open Source Analytics Initiative , 2014, J. Learn. Anal..

[71]  Pallab Paul,et al.  A comparison of concurrent and retrospective verbal protocol analysis. , 2000, The American journal of psychology.

[72]  Yehuda Koren,et al.  Collaborative filtering with temporal dynamics , 2009, KDD.

[73]  Chris Piech,et al.  Learning to Represent Student Knowledge on Programming Exercises Using Deep Learning , 2017, EDM.

[74]  Geoffrey E. Hinton,et al.  Speech recognition with deep recurrent neural networks , 2013, 2013 IEEE International Conference on Acoustics, Speech and Signal Processing.

[75]  Tat-Seng Chua,et al.  Fast Matrix Factorization for Online Recommendation with Implicit Feedback , 2016, SIGIR.

[76]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[77]  Yoav Goldberg,et al.  A Primer on Neural Network Models for Natural Language Processing , 2015, J. Artif. Intell. Res..

[78]  Alvaro Ortigosa,et al.  Recommendation in Higher Education Using Data Mining Techniques , 2009, EDM.

[79]  Arkadiusz Paterek,et al.  Improving regularized singular value decomposition for collaborative filtering , 2007 .

[80]  Yehuda Koren,et al.  Factor in the neighbors: Scalable and accurate collaborative filtering , 2010, TKDD.

[81]  Adam Tauman Kalai,et al.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings , 2016, NIPS.

[82]  B. Zimmerman Self-Regulated Learning and Academic Achievement: An Overview , 1990 .

[83]  Lukás Burget,et al.  Recurrent neural network based language model , 2010, INTERSPEECH.

[84]  Judy Kay,et al.  Intelligent Tutoring Systems , 2000, Lecture Notes in Computer Science.

[85]  Li Chen,et al.  A user-centric evaluation framework for recommender systems , 2011, RecSys '11.

[86]  Michael J. Pazzani,et al.  Content-Based Recommendation Systems , 2007, The Adaptive Web.

[87]  Pasquale Lops,et al.  A Deep Architecture for Content-based Recommendations Exploiting Recurrent Neural Networks , 2017, UMAP.

[88]  Shane Legg,et al.  Human-level control through deep reinforcement learning , 2015, Nature.

[89]  Yong Liu,et al.  Improved Recurrent Neural Networks for Session-based Recommendations , 2016, DLRS@RecSys.

[90]  Sean M. McNee,et al.  Getting to know you: learning new user preferences in recommender systems , 2002, IUI '02.

[91]  Judy Kay,et al.  Open Learner Models , 2010, Advances in Intelligent Tutoring Systems.

[92]  Hugh C. Davis,et al.  Evaluating Three Scrutability and Three Privacy User Privileges for a Scrutable User Modelling Infrastructure , 2009, UMAP.

[93]  Xiangnan He,et al.  A Generic Coordinate Descent Framework for Learning from Implicit Feedback , 2016, WWW.

[94]  Christopher Mazzeo,et al.  Improving Credit Mobility for Community College Transfer Students , 2016 .

[95]  Geoffrey E. Hinton,et al.  Deep Learning , 2015, Nature.

[96]  Nemanja Djuric,et al.  E-commerce in Your Inbox: Product Recommendations at Scale , 2015, KDD.

[97]  Albert T. Corbett,et al.  Cognitive Computer Tutors: Solving the Two-Sigma Problem , 2001, User Modeling.

[98]  Yehuda Koren,et al.  Advances in Collaborative Filtering , 2011, Recommender Systems Handbook.

[99]  Oren Barkan,et al.  ITEM2VEC: Neural item embedding for collaborative filtering , 2016, 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP).

[100]  Robert L. Mercer,et al.  Class-Based n-gram Models of Natural Language , 1992, CL.

[101]  Laurens van der Maaten,et al.  Accelerating t-SNE using tree-based algorithms , 2014, J. Mach. Learn. Res..

[102]  Judy Kay,et al.  Bringing Scrutability to Adaptive Hypertext Teaching , 2000, Intelligent Tutoring Systems.

[103]  Tsvi Kuflik,et al.  Cross-Domain Mediation in Collaborative Filtering , 2007, User Modeling.

[104]  Zachary A. Pardos,et al.  Imputing KCs with Representations of Problem Content and Context , 2017, UMAP.