Improved reviewer assignment based on both word and semantic features

Assigning appropriate reviewers to a manuscript from a pool of candidate reviewers is a common challenge in the academic community. Current word- and semantic-based approaches treat the reviewer assignment problem (RAP) as an information retrieval problem but do not take into account two constraints of the RAP: incompleteness of the reviewer data and interference from nonmanuscript-related papers. In this paper, a word and semantic-based iterative model (WSIM) is proposed to account for the constraints of the RAP by improving the similarity calculations between reviewers and manuscripts. First, we use the improved language model and topic model to extract word features and semantic features to represent reviewers and manuscripts. Second, we use a similarity metric based on the normalized discounted cumulative gain (NDCG) to measure semantic similarity. This metric ignores the probability value (quantitative exact value) of the topic and considers only the ranking (qualitative relevance), thus reducing overfitting to incomplete reviewer data. Finally, we use an iterative model to reduce the interference from nonmanuscript-related papers in the reviewer data. This approach considers the similarity between the manuscript and each of the reviewer’s papers. We evaluate the proposed WSIM on two real datasets and compare its performance to that of seven existing methods. The experimental results show that the WSIM improves the recommendation accuracy by at least 2.5% on the top 20.

[1]  Geneva G. Belford,et al.  Multi-aspect expertise matching for review assignment , 2008, CIKM '08.

[2]  Yiwei Thomas Hou,et al.  The new automated IEEE INFOCOM review assignment system , 2016, IEEE Network.

[3]  Nikos D. Sidiropoulos,et al.  Signal Processing and Optimization Tools for Conference Review and Session Assignment , 2015, IEEE Signal Processing Magazine.

[4]  Peter A. Flach,et al.  Computational support for academic peer review , 2017, Commun. ACM.

[5]  Eunjoo Lee,et al.  Understanding Review Expertise of Developers: A Reviewer Recommendation Approach Based on Latent Dirichlet Allocation , 2018, Symmetry.

[6]  Fan Wang,et al.  Group-to-group reviewer assignment problem , 2013, Comput. Oper. Res..

[7]  Félix Hernández-del-Olmo,et al.  Automatic assignment of reviewers in an online peer assessment task based on social interactions , 2019, Expert Syst. J. Knowl. Eng..

[8]  ChengXiang Zhai,et al.  Constrained multi-aspect expertise matching for committee review assignment , 2009, CIKM.

[9]  Andrew McCallum,et al.  Expertise modeling for matching papers with reviewers , 2007, KDD '07.

[10]  Tom Hunter,et al.  Do Journal Administrators solve the reviewer assignment problem as well as editors? Consideration of reviewer rigour and timeliness , 2019, Learn. Publ..

[11]  Ngoc Thanh Nguyen,et al.  Decision Support System for Assignment of Conference Papers to Reviewers , 2019, ICCCI.

[12]  E. Alper Yildirim,et al.  An alternative polynomial-sized formulation and an optimization based heuristic for the reviewer assignment problem , 2019, Eur. J. Oper. Res..

[13]  William W. Cohen,et al.  Recommendation : A Study in Combining Multiple Information Sources , 2007 .

[14]  ChengXiang Zhai,et al.  Integer linear programming for Constrained Multi-Aspect Committee Review Assignment , 2012, Inf. Process. Manag..

[15]  Nikos Mamoulis,et al.  Weighted Coverage based Reviewer Assignment , 2015, SIGMOD Conference.

[16]  Toby Walsh,et al.  The Conference Paper Assignment Problem: Using Order Weighted Averages to Assign Indivisible Goods , 2017, AAAI.

[17]  Nihar B. Shah,et al.  On Strategyproof Conference Peer Review , 2018, IJCAI.

[18]  Xiaoling Wang,et al.  Time-Aware and Topic-Based Reviewer Assignment , 2017, DASFAA Workshops.

[19]  Devendra K. Tayal,et al.  New method for solving reviewer assignment problem using type-2 fuzzy sets and fuzzy functions , 2013, Applied Intelligence.

[20]  Philippe Rigaux An iterative rating method: application to web-based conference management , 2004, SAC '04.

[21]  Jaroslaw Protasiewicz,et al.  A support system for selection of reviewers , 2014, 2014 IEEE International Conference on Systems, Man, and Cybernetics (SMC).

[22]  Ellen M. Voorhees,et al.  Retrieval evaluation with incomplete information , 2004, SIGIR '04.

[23]  Jakob Nielsen,et al.  Automating the assignment of submitted manuscripts to reviewers , 1992, SIGIR '92.

[24]  Jie Tang,et al.  A Multi-Label Classification Method Using a Hierarchical and Transparent Representation for Paper-Reviewer Recommendation , 2019, ACM Trans. Inf. Syst..

[25]  Richard S. Zemel,et al.  The Toronto Paper Matching System: An automated paper-reviewer assignment system , 2013 .

[26]  Jie Tang,et al.  A novel classification method for paper-reviewer recommendation , 2018, Scientometrics.

[27]  Keun Ho Ryu,et al.  Proposal reviewer recommendation system based on big data for a national research management institute , 2017, J. Inf. Sci..

[28]  Matt J. Kusner,et al.  From Word Embeddings To Document Distances , 2015, ICML.

[29]  Nikos Mamoulis,et al.  A Topic-based Reviewer Assignment System , 2015, Proc. VLDB Endow..

[30]  David Yarowsky,et al.  Taking the load off the conference chairs-towards a digital paper-routing assistant , 1999, EMNLP.

[31]  M. Hasan,et al.  Using Publications and Domain Knowledge to Build Research Profiles: An Application in Automatic Reviewer Assignment , 2007, 2007 International Conference on Information and Communication Technology.

[32]  Jie Tang,et al.  Expertise Matching via Constraint-Based Optimization , 2010, Web Intelligence.

[33]  Roger Wattenhofer,et al.  The Paper Assignment Problem , 2005 .

[34]  Zhengwen Zhang,et al.  Paper review assignment based on human-knowledge network , 2008, 2008 IEEE International Conference on Systems, Man and Cybernetics.

[35]  W. Bruce Croft,et al.  Explainable Product Search with a Dynamic Relation Embedding Model , 2019, ACM Trans. Inf. Syst..

[36]  T. N. Abiodun,et al.  PROPOSED FRAMEWORK FOR A PAPER-REVIEWER ASSIGNMENT SYSTEM USING WORD2VEC , 2017 .

[37]  Yoshua Bengio,et al.  Random Search for Hyper-Parameter Optimization , 2012, J. Mach. Learn. Res..

[38]  Pankaj Vanwari,et al.  A survey on recommendation system for technical paper reviewer assignment , 2017, 2017 International conference of Electronics, Communication and Aerospace Technology (ICECA).

[39]  Stefano Ferilli,et al.  GRAPE: An Expert Review Assignment Component for Scientific Conference Management Systems , 2005, IEA/AIE.

[40]  Stefano Ferilli,et al.  Automatic Topics Identification for Reviewer Assignment , 2006, IEA/AIE.

[41]  Yuming Jiang,et al.  Deep semantics inspection over big network data at wire speed , 2016, IEEE Network.

[42]  Cheng Long,et al.  On Good and Fair Paper-Reviewer Assignment , 2013, 2013 IEEE 13th International Conference on Data Mining.

[43]  Jian Ma,et al.  An intelligent decision support approach for reviewer assignment in R&D project selection , 2016, Comput. Ind..

[44]  Toyohide Watanabe,et al.  Automatic Paper-to-reviewer Assignment, based on the Matching Degree of the Reviewers , 2013, KES.

[45]  Charles L. A. Clarke,et al.  Information Retrieval - Implementing and Evaluating Search Engines , 2010 .

[46]  Michael J. Pazzani,et al.  Mining for proposal reviewers: lessons learned at the national science foundation , 2006, KDD '06.

[47]  Craig Boutilier,et al.  A Framework for Optimizing Paper Matching , 2011, UAI.

[48]  Witold Pedrycz,et al.  A recommender system of reviewers and experts in reviewing problems , 2016, Knowl. Based Syst..

[49]  Bo Gao,et al.  On optimization of expertise matching with various constraints , 2012, Neurocomputing.

[50]  Peter A. Flach,et al.  Novel tools to streamline the conference review process: experiences from SIGKDD'09 , 2010, SKDD.

[51]  Fan Wang,et al.  A Survey on Reviewer Assignment Problem , 2008, IEA/AIE.

[52]  James Allan,et al.  A comparison of statistical significance tests for information retrieval evaluation , 2007, CIKM '07.