Dual Attention Model for Citation Recommendation

Based on an exponentially increasing number of academic articles, discovering and citing comprehensive and appropriate resources has become a non-trivial task. Conventional citation recommender methods suffer from severe information loss. For example, they do not consider the section of the paper that the user is writing and for which they need to find a citation, the relatedness between the words in the local context (the text span that describes a citation), or the importance on each word from the local context. These shortcomings make such methods insufficient for recommending adequate citations to academic manuscripts. In this study, we propose a novel embedding-based neural network called "dual attention model for citation recommendation (DACR)" to recommend citations during manuscript preparation. Our method adapts embedding of three dimensions of semantic information: words in the local context, structural contexts, and the section on which a user is working. A neural network is designed to maximize the similarity between the embedding of the three input (local context words, section and structural contexts) and the target citation appearing in the context. The core of the neural network is composed of self-attention and additive attention, where the former aims to capture the relatedness between the contextual words and structural context, and the latter aims to learn the importance of them. The experiments on real-world datasets demonstrate the effectiveness of the proposed approach.

[1]  Nitish Srivastava,et al.  Learning Generative Models with Visual Attention , 2013, NIPS.

[2]  Quoc V. Le,et al.  Distributed Representations of Sentences and Documents , 2014, ICML.

[3]  Qiang Ma,et al.  DocCit2Vec: Citation Recommendation via Embedding of Content and Structural Contexts , 2020, IEEE Access.

[4]  Geoffrey E. Hinton,et al.  On the importance of initialization and momentum in deep learning , 2013, ICML.

[5]  Jeffrey Dean,et al.  Efficient Estimation of Word Representations in Vector Space , 2013, ICLR.

[6]  Marco Gori,et al.  Recommender Systems : A Random-Walk Based Approach , 2006 .

[7]  Katherine McDonough,et al.  cite2vec: Citation-Driven Document Exploration via Word Embeddings , 2017, IEEE Transactions on Visualization and Computer Graphics.

[8]  Shuming Shi,et al.  hyperdoc2vec: Distributed Representations of Hypertext Documents , 2018, ACL.

[9]  Daniel Kifer,et al.  Context-aware citation recommendation , 2010, WWW '10.

[10]  Qiang Ma,et al.  Citation Recommendations Considering Content and Structural Context Embedding , 2020, 2020 IEEE International Conference on Big Data and Smart Computing (BigComp).

[11]  Victor Anthony Arrascue Ayala,et al.  PubRec: Recommending Publications Based on Publicly Available Meta-Data , 2015, LWA.

[12]  Geoffrey E. Hinton,et al.  Layer Normalization , 2016, ArXiv.

[13]  Xiang Cheng,et al.  Conference Paper Recommendation for Academic Conferences , 2018, IEEE Access.

[14]  Lee M. Seversky,et al.  2 vec : Citation-Driven Document Exploration via Word Embeddings , 2016 .

[15]  Ümit V. Çatalyürek,et al.  Towards a personalized, scalable, and exploratory academic recommendation service , 2013, 2013 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2013).

[16]  Suyu Ge,et al.  Neural News Recommendation with Multi-Head Self-Attention , 2019, EMNLP.

[17]  Min-Yen Kan,et al.  Logical Structure Recovery in Scholarly Articles with Rich Document Features , 2010, Int. J. Digit. Libr. Syst..

[18]  Natalia Gimelshein,et al.  PyTorch: An Imperative Style, High-Performance Deep Learning Library , 2019, NeurIPS.

[19]  Cornelia Caragea,et al.  Can't see the forest for the trees?: a citation recommendation system , 2013, JCDL '13.

[20]  Jian Pei,et al.  Citation recommendation without author supervision , 2011, WSDM '11.

[21]  Sean M. McNee,et al.  On the recommending of citations for research papers , 2002, CSCW '02.

[22]  C. Lee Giles,et al.  ParsCit: an Open-source CRF Reference String Parsing Package , 2008, LREC.

[23]  Jeffrey Dean,et al.  Distributed Representations of Words and Phrases and their Compositionality , 2013, NIPS.

[24]  Nitish Srivastava,et al.  Improving neural networks by preventing co-adaptation of feature detectors , 2012, ArXiv.

[25]  Erik Saule,et al.  An Analysis of Citation Recommender Systems: Beyond the Obvious , 2017, ASONAM.

[26]  Petr Sojka,et al.  Software Framework for Topic Modelling with Large Corpora , 2010 .

[27]  Lukasz Kaiser,et al.  Attention is All you Need , 2017, NIPS.

[28]  Erik Saule,et al.  Local Is Good: A Fast Citation Recommendation Approach , 2018, ECIR.

[29]  Ramón Fernández Astudillo,et al.  Not All Contexts Are Created Equal: Better Word Representations with Variable Attention , 2015, EMNLP.

[30]  Ming-Wei Chang,et al.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding , 2019, NAACL.