Near-Optimal Reviewer Splitting in Two-Phase Paper Reviewing and Conference Experiment Design

Many scientific conferences employ a two-phase paper review process, where some papers are assigned additional reviewers after the initial reviews are submitted. Many conferences also design and run experiments on their paper review process, where some papers are assigned reviewers who provide reviews under an experimental condition. In this paper, we consider the question: how should reviewers be divided between phases or conditions in order to maximize total assignment similarity? We make several contributions towards answering this question. First, we prove that when the set of papers requiring additional review is unknown, a simplified variant of this problem is NP-hard. Second, we empirically show that across several datasets pertaining to real conference data, dividing reviewers between phases/conditions uniformly at random allows an assignment that is nearly as good as the oracle optimal assignment. This uniformly random choice is practical for both the two-phase and conference experiment design settings. Third, we provide explanations of this phenomenon by providing theoretical bounds on the suboptimality of this random strategy under certain natural conditions. From these easily-interpretable conditions, we provide actionable insights to conference program chairs about whether a random reviewer split is suitable for their conference.

[1]  Nihar B. Shah,et al.  A large scale randomized controlled trial on herding in peer-review discussions , 2020, PloS one.

[2]  Euiwoong Lee,et al.  Maximum Matching in the Online Batch-arrival Model , 2017, IPCO.

[3]  Omer Lev,et al.  Strategyproof peer selection using randomization, partitioning, and apportionment , 2016, Artif. Intell..

[4]  Tim Kraska,et al.  The SIGMOD 2019 Research Track Reviewing System , 2019, SGMD.

[5]  Cheng Long,et al.  On Good and Fair Paper-Reviewer Assignment , 2013, 2013 IEEE 13th International Conference on Data Mining.

[6]  Aravind Srinivasan,et al.  Allocation Problems in Ride-sharing Platforms , 2017, AAAI.

[7]  Ryan O'Donnell,et al.  Lower Bounds for Testing Function Isomorphism , 2010, 2010 IEEE 25th Annual Conference on Computational Complexity.

[8]  Jie Tang,et al.  Expertise Matching via Constraint-Based Optimization , 2010, Web Intelligence.

[9]  Jan Vondrák,et al.  Submodular maximization by simulated annealing , 2010, SODA '11.

[10]  Nihar B. Shah,et al.  Prior and Prejudice , 2020, Proc. ACM Hum. Comput. Interact..

[11]  Peter A. Flach,et al.  Novel tools to streamline the conference review process: experiences from SIGKDD'09 , 2010, SKDD.

[12]  Richard S. Zemel,et al.  The Toronto Paper Matching System: An automated paper-reviewer assignment system , 2013 .

[13]  Isabelle Guyon,et al.  Design and Analysis of the NIPS 2016 Review Process , 2017, J. Mach. Learn. Res..

[14]  Andrew McCallum,et al.  Paper Matching with Local Fairness Constraints , 2019, KDD.

[15]  Richard M. Karp,et al.  Reducibility Among Combinatorial Problems , 1972, 50 Years of Integer Programming.

[16]  Nihar B. Shah.  KDD 2021 Tutorial on Systemic Challenges and Solutions on Bias and Unfairness in Peer Review , 2021, KDD.

[17]  Nihar B. Shah,et al.  A SUPER* Algorithm to Optimize Paper Bidding in Peer Review , 2020, UAI.

[18]  Noga Alon,et al.  Sum of us: strategyproof selection from the selectors , 2009, TARK XIII.

[19]  David J. DeWitt,et al.  Impact of double-blind reviewing on SIGMOD publication rates , 2006, SGMD.

[20]  Nihar B. Shah,et al.  Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment , 2020, AAAI.

[21]  Aranyak Mehta,et al.  Online Stochastic Matching: Beating 1-1/e , 2009, 2009 50th Annual IEEE Symposium on Foundations of Computer Science.

[22]  Yoav Zemel,et al.  Statistical Aspects of Wasserstein Distances , 2018, Annual Review of Statistics and Its Application.

[23]  Nihar B. Shah,et al.  Uncovering Latent Biases in Text: Method and Application to Peer Review , 2020, AAAI.

[24]  Aravind Srinivasan,et al.  New Algorithms, Better Bounds, and a Novel Model for Online Stochastic Matching , 2016, ESA.

[25]  W. E. Kerzendorf,et al.  The Distributed Peer Review Experiment , 2019 .

[26]  Amin Saberi,et al.  Two-stage Stochastic Matching with Application to Ride Hailing , 2021, SODA.

[27]  Kurt Mehlhorn,et al.  Assigning Papers to Referees , 2009, Algorithmica.

[28]  Nihar B. Shah,et al.  On Testing for Biases in Peer Review , 2019, NeurIPS.

[29]  Nicholas Mattei,et al.  A Market-Inspired Bidding Scheme for Peer Review Paper Assignment , 2021, AAAI.

[30]  Joseph Naor,et al.  Submodular Maximization with Cardinality Constraints , 2014, SODA.

[31]  Nasir D. Memon,et al.  A robust model for paper reviewer assignment , 2014, RecSys '14.

[32]  Nihar B. Shah,et al.  Mitigating Manipulation in Peer Review via Randomized Reviewer Assignments , 2020, NeurIPS.

[33]  Andrew J. Schaefer,et al.  A factor 1/2 approximation algorithm for two-stage stochastic matching problems , 2006, Eur. J. Oper. Res..

[34]  Guillaume Cabanac,et al.  Capitalizing on order effects in the bids of peer-reviewed conferences to secure reviews by expert referees , 2013, J. Assoc. Inf. Sci. Technol..

[35]  Geneva G. Belford,et al.  Multi-aspect expertise matching for review assignment , 2008, CIKM '08.

[36]  H. Robbins.  A Remark on Stirling’s Formula , 1955 .

[37]  Johan Bollen,et al.  An algorithm to determine peer-reviewers , 2006, CIKM '08.

[38]  Andrew McCallum,et al.  Expertise modeling for matching papers with reviewers , 2007, KDD '07.

[39]  Min Zhang,et al.  Reviewer bias in single- versus double-blind peer review , 2017, Proceedings of the National Academy of Sciences.

[40]  W. Hoeffding.  Probability Inequalities for sums of Bounded Random Variables , 1963 .

[41]  Vahab S. Mirrokni,et al.  Maximizing Non-Monotone Submodular Functions , 2011, 48th Annual IEEE Symposium on Foundations of Computer Science (FOCS'07).

[42]  Thomas Rothvoß.  A direct proof for Lovett's bound on the communication complexity of low rank matrices , 2014, ArXiv.

[43]  Craig Boutilier,et al.  A Framework for Optimizing Paper Matching , 2011, UAI.

[44]  Olivier Spanjaard,et al.  Two-stage stochastic matching and spanning tree problems: Polynomial instances and approximation , 2010, Eur. J. Oper. Res..

[45]  Judy Goldsmith,et al.  The AI conference paper assignment problem , 2007, AAAI 2007.

[46]  S. Ceci,et al.  Peer review--a study of reliability. , 1982, Change.

[47]  Baruch Schieber,et al.  Generalized Assignment via Submodular Optimization with Reserved Capacity , 2019, ESA.

[48]  Anthony K. H. Tung.  Impact of double blind reviewing on SIGMOD publication: a more detail analysis , 2006, SGMD.

[49]  Nihar B. Shah,et al.  On Strategyproof Conference Peer Review , 2018, IJCAI.

[50]  Nihar B. Shah,et al.  PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review , 2018, ALT.

[51]  Camillo J. Taylor,et al.  On the Optimal Assignment of Conference Papers to Reviewers , 2008 .

[52]  T. Banta,et al.  Change : The Magazine of Higher Learning , 2010 .

[53]  Chun-Hung Chen,et al.  Convergence Properties of Two-Stage Stochastic Programming , 2000 .

[54]  Mitchell J. Nathan,et al.  'Your comments are meaner than your score': score calibration talk influences intra- and inter-panel variability during scientific grant peer review. , 2017, Research Evaluation.

[55]  Guillaume Cabanac,et al.  Expert suggestion for conference program committees , 2017, 2017 11th International Conference on Research Challenges in Information Science (RCIS).

[56]  Ariel D. Procaccia,et al.  Dynamic Matching via Weighted Myopia with Application to Kidney Exchange , 2012, AAAI.

[57]  J. Scott Armstrong,et al.  Unintelligible Management Research and Academic Prestige , 1980 .

[58]  R. Tyrrell Rockafellar,et al.  Asymptotic Theory for Solutions in Statistical Estimation and Stochastic Programming , 1993, Math. Oper. Res..

[59]  Eli Upfal,et al.  Commitment under uncertainty: Two-stage stochastic matching problems , 2007, Theor. Comput. Sci..