Reinforcement Learning Page Prediction for Hierarchically Ordered Municipal Websites

Public websites offer information on a variety of topics and services and are accessed by users with varying skills to browse the kind of electronic document repositories. However, the complex website structure and diversity of web browsing behavior create a challenging task for click prediction. This paper presents the results of a novel reinforcement learning approach to model user browsing patterns in a hierarchically ordered municipal website. We study how accurate predictor the browsing history is, when the target pages are not immediate next pages pointed by hyperlinks, but appear a number of levels down the hierarchy. We compare traditional type of baseline classifiers’ performance against our reinforcement learning-based training algorithm.

[1]  Dawei Yin,et al.  Pseudo Dyna-Q: A Reinforcement Learning Framework for Interactive Recommendation , 2020, WSDM.

[2]  Dietmar Jannach,et al.  Sequence-Aware Recommender Systems , 2018, UMAP.

[3]  Jugal K. Kalita,et al.  A Survey of the Usages of Deep Learning for Natural Language Processing , 2018, IEEE Transactions on Neural Networks and Learning Systems.

[4]  Lina Yao,et al.  Deep Learning Based Recommender System , 2017, ACM Comput. Surv..

[5]  Kostas Stefanidis,et al.  On Measuring Popularity Bias in Collaborative Filtering Data , 2020, EDBT/ICDT Workshops.

[6]  Michalis Vazirgiannis,et al.  Web path recommendations based on page ranking and Markov models , 2005, WIDM '05.

[7]  Cheng Guo,et al.  Entity Embeddings of Categorical Variables , 2016, ArXiv.

[8]  Jure Leskovec,et al.  node2vec: Scalable Feature Learning for Networks , 2016, KDD.

[9]  Jimeng Sun,et al.  Hierarchical Reinforcement Learning for Course Recommendation in MOOCs , 2019, AAAI.

[10]  Erik B. Sudderth.  Introduction to statistical machine learning , 2016 .

[11]  Svetlana Peltsverger,et al.  Web Analytics Overview , 2015 .

[12]  Kostas Stefanidis,et al.  On mitigating popularity bias in recommendations via variational autoencoders , 2021, SAC.

[13]  Satoshi Sekine,et al.  A survey of named entity recognition and classification , 2007 .

[14]  Gholamreza Haffari,et al.  Reasoning Like Human: Hierarchical Reinforcement Learning for Knowledge Graph Reasoning , 2020, IJCAI.

[15]  Steven Skiena,et al.  DeepWalk: online learning of social representations , 2014, KDD.

[16]  Taghi M. Khoshgoftaar,et al.  A Survey of Collaborative Filtering Techniques , 2009, Adv. Artif. Intell..

[17]  Fabien L. Gandon,et al.  Proceedings of the 2018 World Wide Web Conference , 2018 .

[18]  Neetu Sardana,et al.  Web navigation prediction based on dynamic threshold heuristics , 2020 .

[19]  Terence Parr,et al.  The Matrix Calculus You Need For Deep Learning , 2018, ArXiv.

[20]  Kostas Stefanidis,et al.  Enhancing Long Term Fairness in Recommendations with Variational Autoencoders , 2019, MEDES.

[21]  Grigorios Tsoumakas,et al.  Mining Multi-label Data , 2010, Data Mining and Knowledge Discovery Handbook.

[22]  Caroline Herssens,et al.  Knowledge-Based Recommendation Systems: A Survey , 2014, Int. J. Intell. Inf. Technol..

[23]  Martha Larson,et al.  Collaborative Filtering beyond the User-Item Matrix , 2014, ACM Comput. Surv..

[24]  Min-Ling Zhang,et al.  A Review on Multi-Label Learning Algorithms , 2014, IEEE Transactions on Knowledge and Data Engineering.

[25]  Heng-Tze Cheng,et al.  Wide & Deep Learning for Recommender Systems , 2016, DLRS@RecSys.

[26]  Chang Zhou,et al.  Scalable Graph Embedding for Asymmetric Proximity , 2017, AAAI.

[27]  Tat-Seng Chua,et al.  Neural Collaborative Filtering , 2017, WWW.

[28]  Yu Lei,et al.  Reinforcement Learning based Recommendation with Graph Convolutional Q-network , 2020, SIGIR.

[29]  Yi Tay,et al.  Deep Learning based Recommender System: A Survey and New Perspectives , 2018 .

[30]  Daniel Kudenko,et al.  Deep Reinforcement Learning with Graph-based State Representations , 2020, ArXiv.

[31]  Jure Leskovec,et al.  Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation , 2018, NeurIPS.

[32]  Yixin Cao,et al.  Reinforced Negative Sampling over Knowledge Graph for Recommendation , 2020, WWW.

[33]  Sheldon M. Ross,et al.  Stochastic Processes , 2018, Gauge Integral Structures for Stochastic Calculus and Quantum Electrodynamics.

[34]  Lior Rokach,et al.  Data Mining And Knowledge Discovery Handbook , 2005 .

[35]  Yong Yu,et al.  Large-scale Interactive Recommendation with Tree-structured Policy Gradient , 2018, AAAI.

[36]  Rubing Duan,et al.  Neural Modeling of Buying Behaviour for E-Commerce from Clicking Patterns , 2015, RecSys Challenge.

[37]  Christophe Prieur,et al.  Role of the Website Structure in the Diversity of Browsing Behaviors , 2019, HT.

[38]  Gábor Petneházi,et al.  Recurrent Neural Networks for Time Series Forecasting , 2018, ArXiv.

[39]  Padraig Corcoran,et al.  Deep Q-Learning for Directed Acyclic Graph Generation , 2019, ArXiv.

[40]  Yiwei Zhang,et al.  Reinforcement Mechanism Design for e-commerce , 2017, WWW.

[41]  Tat-Seng Chua,et al.  Attentional Factorization Machines: Learning the Weight of Feature Interactions via Attention Networks , 2017, IJCAI.

[42]  T. Peráček,et al.  User-Engagement Score and SLIs/SLOs/SLAs Measurements Correlation of E-Business Projects Through Big Data Analysis , 2020, Applied Sciences.

[43]  Kostas Stefanidis,et al.  Recommendations beyond the ratings matrix , 2016, DDI@WebSci.

[44]  Saeed Shiry Ghidary,et al.  Usage-based web recommendations: a reinforcement learning approach , 2007, RecSys '07.

[45]  Michael Scholz.  R Package clickstream: Analyzing Clickstream Data with Markov Chains , 2016 .

[46]  Evaggelia Pitoura,et al.  Fair sequential group recommendations , 2020, SAC.

[47]  Yunming Ye,et al.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction , 2017, IJCAI.