Sets2Sets: Learning from Sequential Sets with Neural Networks

Given past sequential sets of elements, predicting the subsequent sets of elements is an important problem in different domains. With the past orders of customers given, predicting the items that are likely to be bought in their following orders can provide information about the future purchase intentions. With the past clinical records of patients at each visit to the hospitals given, predicting the future clinical records in the subsequent visits can provide information about the future disease progression. These useful information can help to make better decisions in different domains. However, existing methods have not studied this problem well. In this paper, we formulate this problem as a sequential sets to sequential sets learning problem. We propose an end-to-end learning approach based on an encoder-decoder framework to solve the problem. In the encoder, our approach maps the set of elements at each past time step into a vector. In the decoder, our method decodes the set of elements at each subsequent time step from the vectors with a set-based attention mechanism. The repeated elements pattern is also considered in our method to further improve the performance. In addition, our objective function addresses the imbalance and correlation existing among the predicted elements. The experimental results on three real-world data sets showthat our method outperforms the best performance of the compared methods with respect to recall and person-wise hit ratio by 2.7-20.6% and 2.1-26.3%, respectively. Our analysis also shows that our decoder has good generalization to output sequential sets that are even longer than the output of training instances.

[1]  Jimmy Ba,et al.  Adam: A Method for Stochastic Optimization , 2014, ICLR.

[2]  Xiangnan He,et al.  Attentive Collaborative Filtering: Multimedia Recommendation with Item- and Component-Level Attention , 2017, SIGIR.

[3]  Alhassan G. Abdul-Muhmin.  Repeat Purchase Intentions in Online Shopping: The Role of Satisfaction, Attitude, and Online Retailers' Performance , 2010 .

[4]  Walter F. Stewart,et al.  Doctor AI: Predicting Clinical Events via Recurrent Neural Networks , 2015, MLHC.

[5]  George Karypis,et al.  Cumulative Knowledge-based Regression Models for Next-term Grade Prediction , 2017, SDM.

[6]  Lars Schmidt-Thieme,et al.  Factorizing personalized Markov chains for next-basket recommendation , 2010, WWW '10.

[7]  Samy Bengio,et al.  Show and tell: A neural image caption generator , 2014, 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[8]  Min-Ling Zhang,et al.  A Review on Multi-Label Learning Algorithms , 2014, IEEE Transactions on Knowledge and Data Engineering.

[9]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[10]  Yoshua Bengio,et al.  Neural Machine Translation by Jointly Learning to Align and Translate , 2014, ICLR.

[11]  Valentin Flunkert,et al.  DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks , 2017, International Journal of Forecasting.

[12]  Shanshan Zhang,et al.  Interpretable Representation Learning for Healthcare via Capturing Disease Progression through Time , 2018, KDD.

[13]  Ravi Kumar,et al.  Sequences of Sets , 2018, KDD.

[14]  Pengfei Wang,et al.  Learning Hierarchical Representation Model for NextBasket Recommendation , 2015, SIGIR.

[15]  Yulia Tsvetkov,et al.  Problems With Evaluation of Word Embeddings Using Word Similarity Tasks , 2016, RepEval@ACL.

[16]  Hagen Soltau,et al.  Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition , 2016, INTERSPEECH.

[17]  Dimitris N. Metaxas,et al.  Addressing Imbalance in Multi-Label Classification Using Structured Hellinger Forests , 2017, AAAI.

[18]  T. Orchard,et al.  Cognitive dysfunction in adults with Type 1 (insulin-dependent) diabetes mellitus of long duration: effects of recurrent hypoglycaemia and other chronic complications , 1993, Diabetologia.

[19]  Samy Bengio,et al.  Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks , 2015, NIPS.

[20]  Quoc V. Le,et al.  Sequence to Sequence Learning with Neural Networks , 2014, NIPS.

[21]  Gwilym M. Jenkins,et al.  Time series analysis, forecasting and control , 1971 .

[22]  Diyar Akay,et al.  Comparison of direct and iterative artificial neural network forecast approaches in multi-periodic time series forecasting , 2009, Expert Syst. Appl..

[23]  Tao Chen,et al.  TriRank: Review-aware Explainable Recommendation by Modeling Aspects , 2015, CIKM.

[24]  Yoshua Bengio,et al.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches , 2014, SSST@EMNLP.

[25]  P. Young,et al.  Time series analysis, forecasting and control , 1972, IEEE Transactions on Automatic Control.

[26]  Feng Yu,et al.  A Dynamic Recurrent Model for Next Basket Recommendation , 2016, SIGIR.

[27]  Xiangnan He,et al.  NAIS: Neural Attentive Item Similarity Model for Recommendation , 2018, IEEE Transactions on Knowledge and Data Engineering.

[28]  Tat-Seng Chua,et al.  Neural Graph Collaborative Filtering , 2019, SIGIR.

[29]  Daniel T. Larose,et al.  Discovering Knowledge in Data: An Introduction to Data Mining , 2005 .

[30]  Zhi-Hua Zhou,et al.  Multilabel Neural Networks with Applications to Functional Genomics and Text Categorization , 2006, IEEE Transactions on Knowledge and Data Engineering.

[31]  Alex Graves,et al.  Recurrent Models of Visual Attention , 2014, NIPS.

[32]  Evangelia Christakopoulou,et al.  Local Latent Space Models for Top-N Recommendation , 2018, KDD.

[33]  Andrew McCallum,et al.  Collective multi-label classification , 2005, CIKM '05.

[34]  Russell L. Purvis,et al.  Forecasting the NYSE composite index with technical analysis, pattern recognizer, neural network, and genetic algorithm: a case study in romantic decision support , 2002, Decis. Support Syst..