A frequently encountered problem in decision making is the following review problem: review a large number of objects and select a small number of the best ones. An example is selecting conference papers from a large number of submissions. This problem involves two sub-problems: assigning reviewers to each object, and summarizing reviewers’ scores into an overall score that supposedly reflects the quality of an object. In this paper, we address the score summarization sub-problem for the scenario where a small number of reviewers evaluate each object. Simply averaging the scores may not work as even a single reviewer could influence the average significantly. We recognize that reviewers are not necessarily on an equal ground and propose the notion of “leniency” to model this difference of reviewers. Two insights underpin our approach: (1) the “leniency” of a reviewer depends on how s/he evaluates objects as well as on how other reviewers evaluate the same set of objects, (2) the “leniency” of a reviewer and the “quality” of objects evaluated exhibit a mutual dependency relationship. These insights motivate us to develop a model that solves both “leniency” and “quality” simultaneously. We study the effectiveness of this model on a real-life dataset.
[1]
Michael J. Pazzani,et al.
Mining for proposal reviewers: lessons learned at the national science foundation
,
2006,
KDD '06.
[2]
G. Grimmett,et al.
Probability and random processes
,
2002
.
[3]
Gene H. Golub,et al.
Matrix computations
,
1983
.
[4]
H. Arkes.
The Nonuse of Psychological Research at Two Federal Agencies
,
2003,
Psychological science.
[5]
H. Marsh,et al.
Making students' evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility.
,
1997
.
[6]
Ke Wang,et al.
Bias and controversy: beyond the statistical deviation
,
2006,
KDD '06.
[7]
James Geller.
Challenge: How IJCAI 1999 can Prove Value of AI by Using AI
,
1997,
IJCAI.
[8]
H. Anton,et al.
Elementary linear algebra with applications
,
1987
.
[9]
Rajeev Motwani,et al.
The PageRank Citation Ranking : Bringing Order to the Web
,
1999,
WWW 1999.
[10]
Jakob Nielsen,et al.
Automating the assignment of submitted manuscripts to reviewers
,
1992,
SIGIR '92.