On Good and Fair Paper-Reviewer Assignment

Peer review has become the most common practice for judging papers submitted to a conference for decades. An extremely important task involved in peer review is to assign submitted papers to reviewers with appropriate expertise which is referred to as paper-reviewer assignment. In this paper, we study the paper-reviewer assignment problem from both the goodness aspect and the fairness aspect. For the goodness aspect, we propose to maximize the topic coverage of the paper-reviewer assignment. This objective is new and the problem based on this objective is shown to be NP-hard. To solve this problem efficiently, we design an approximate algorithm which gives a 1/3-approximation. For the fairness aspect, we perform a detailed study on conflict-of-interest (COI) types and discuss several issues related to using COI, which, we hope, can raise some open discussions among researchers on the COI study. Finally, we conducted experiments on real datasets which verified the effectiveness of our algorithm and also revealed some interesting results of COI.

[1]  Jakob Nielsen,et al.  Automating the assignment of submitted manuscripts to reviewers , 1992, SIGIR '92.

[2]  D. Hochbaum.  Approximating covering and packing problems: set cover, vertex cover, independent set, and related problems , 1996 .

[3]  Naren Ramakrishnan,et al.  Recommender systems for the conference paper assignment problem , 2009, RecSys '09.

[4]  Craig Boutilier,et al.  A Framework for Optimizing Paper Matching , 2011, UAI.

[5]  Michael McGill,et al.  Introduction to Modern Information Retrieval , 1983 .

[6]  Stefano Ferilli,et al.  Automatic Topics Identification for Reviewer Assignment , 2006, IEA/AIE.

[7]  William W. Cohen,et al.  Recommending Papers by Mining the Web , 1999 .

[8]  Jie Tang,et al.  Expertise Matching via Constraint-Based Optimization , 2010, Web Intelligence.

[9]  Geneva G. Belford,et al.  Multi-aspect expertise matching for review assignment , 2008, CIKM '08.

[10]  Jan Vondrák,et al.  Maximizing a Submodular Set Function Subject to a Matroid Constraint (Extended Abstract) , 2007, IPCO.

[11]  Andrew McCallum,et al.  Expertise modeling for matching papers with reviewers , 2007, KDD '07.

[12]  ChengXiang Zhai,et al.  Constrained multi-aspect expertise matching for committee review assignment , 2009, CIKM.

[13]  M. L. Fisher,et al.  An analysis of approximations for maximizing submodular set functions—I , 1978, Math. Program..

[14]  Camillo J. Taylor,et al.  On the Optimal Assignment of Conference Papers to Reviewers , 2008 .

[15]  Michael J. Pazzani,et al.  Mining for proposal reviewers: lessons learned at the national science foundation , 2006, KDD '06.

[16]  David Hartvigsen,et al.  The Conference Paper‐Reviewer Assignment Problem* , 1999 .

[17]  Kurt Mehlhorn,et al.  Assigning Papers to Referees , 2009, Algorithmica.

[18]  M. Hasan,et al.  Using Publications and Domain Knowledge to Build Research Profiles: An Application in Automatic Reviewer Assignment , 2007, 2007 International Conference on Information and Communication Technology.

[19]  Salem Benferhat,et al.  Conference paper assignment , 2001, Int. J. Intell. Syst..

[20]  Johan Bollen,et al.  An algorithm to determine peer-reviewers , 2006, CIKM '08.

[21]  Juan Julián Merelo Guervós,et al.  Conference Paper Assignment Using a Combined Greedy/Evolutionary Algorithm , 2004, PPSN.

[22]  Djoerd Hiemstra,et al.  Modeling multi-step relevance propagation for expert finding , 2008, CIKM '08.

[23]  M. de Rijke,et al.  Expertise Retrieval , 2012, Found. Trends Inf. Retr..